JP2014071277A - Head-mounted display device, method of actuating the same and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve visibility when an observer views a reference content in a monocular type head-mounted display enabling the observer to observe at least one object and at least one reference content in an object space in association with each other.SOLUTION: In order to determine which eye of a left eye and a right eye of the observer is an observation eye, a position of the observation eye is detected, and on the basis of the detection result, at least one reference content L or R is displayed at a position shifted from an image of at least one object A observed by the observer in a first direction away from a non-observation eye serving as an eye not used in observation of both eyes of the observer on an image display surface of the head-mounted display.

Description

  The present invention relates to a monocular head-mounted display that displays an image to an observer by projecting image light onto an observation eye that is one eye of the observer while being mounted on the observer's head. In particular, the present invention relates to a technique that enables an observer to observe by associating at least one object in a target space with at least one reference content.

  A head-mounted display (hereinafter abbreviated as “HMD”) that displays an image to the observer while being mounted on the observer's head has already been known. According to the HMD, an observer can directly observe an image without using a screen on which the image is projected.

  This HMD is classified into, for example, a spatial modulation type and a scanning type from the viewpoint of a method of forming image light. The spatial modulation type is a system in which the image light is formed from light emitted from a light source using a spatial modulation element (for example, liquid crystal or organic EL (electroluminescence)) that operates according to an image signal. On the other hand, the scanning type is a system in which the image light is formed by scanning light emitted from a light source and having intensity corresponding to the image signal.

  In addition, this HMD uses both the eyes of the observer as the observation eyes from the viewpoint of whether the eyes of the observer observing the image are both eyes or the eyes of the observer. Are classified into a binocular system that displays an image to the observer by projecting and a monocular system that displays the image to the observer by projecting image light to only one eye. Is done.

  The HMD is classified into, for example, a see-through type and a sealed type from the viewpoint of a method in which an observer observes an image. In the see-through method, image light representing an image according to an image signal and external light from an actual external environment (an example of the “target space”) positioned in front of the observer are superimposed and incident on the eyes of the observer. In this way, the observer can observe an image corresponding to the image signal superimposed on an image of the actual outside world. On the other hand, the sealed type is a system that blocks the incident light of the external light representing the actual external environment and allows the observer to mainly observe only the image corresponding to the image signal.

  When the latter sealed type is adopted, it is possible to design the HMD so that the image corresponding to the image signal does not include the image of the target space. In this case, the observer cannot observe the image of the target space, and observes the image corresponding to the image signal alone. However, it is also possible to design the HMD so as to capture an image of the target space, combine the captured image with another image corresponding to the image signal, and display the combined image to the observer. . In this case, the observer can observe another image superimposed on the image of the target space, as in the case of adopting the see-through method. Hereinafter, for convenience of explanation, this method is referred to as a “video see-through type”, while the above-described see-through type is referred to as an “optical see-through type” to distinguish between them.

  There is already a technology for augmenting and expanding the real world perceived by humans using a computer, which is called augmented reality (AR). One conventional example of a technique for realizing this augmented reality using an optical see-through type or video see-through type HMD is disclosed in Patent Document 1.

  According to the technique disclosed in Patent Document 1, while an observer observes a virtual object existing in a virtual space (another example of the “target space”), the observation position of the virtual object is set. In association with each other, reference content (for example, annotation) related to the virtual object is displayed. According to this technology, it becomes possible for the observer to visually and automatically acquire information related to the virtual object while observing the virtual object, and the convenience of the observer is improved.

JP 2005-038008 A

  According to the technique disclosed in Patent Document 1, when the observer observes the only virtual object, when the observer gazes at the virtual object for a certain period of time, in the vicinity of the observation position of the virtual object, Annotation as reference content related to the virtual object is automatically displayed. At this time, in order to make the focus on the annotation coincide with the focus on the virtual object, the depth from the observer's viewpoint position to the annotation is almost the same as the depth from the viewpoint position to the virtual object.

  On the other hand, the present inventors have studied monocular HMD, and as a result, in order to improve the visibility when the viewer visually recognizes the reference content, the displayed reference content is observed. I realized it was important to optimize the relative position of the object to the image.

  Specifically, the inventors of the present invention, on the image display surface of the monocular HMD, as a relative position of the reference content with respect to the object image, a position deviated from the object image in a direction away from the non-observing eye. And an experiment for evaluating the visibility of the observer with respect to the position deviated in the direction approaching the non-observing eye. As a result, the present inventors displayed the reference content at a position away from the object image in a direction away from the non-observing eye, regardless of whether the HMD is an optical see-through type or a video see-through type. In this case, the viewer's line of sight when the observing eye observes the reference content does not have to be greatly inclined to the non-observing eye side, as compared with the case where it is displayed at a position away from the non-observing eye. Obtained the improvement.

  In addition, the present inventors are particularly the case where the monocular HMD is an optical see-through type, and an optical element for realizing optical see-through at a position near the non-observing eye in the housing of the HMD ( For example, when a half mirror (to be described later) is attached in an exposed state, when the reference content is displayed at a position away from the object image in a direction approaching the non-observation eye, the observation eye observes the reference content. It was noticed that there was a high possibility that the observer's line of sight would intersect the housing or the optical element. If the line of sight intersects with the housing or the optical element, the observation eye cannot normally observe the displayed reference content, and the observer may feel uncomfortable. is there.

  With the knowledge described above as a background, the present invention relates to a monocular HMD that enables an observer to observe at least one object in a target space and at least one reference content in association with each other. The object is to improve the visibility when viewing the reference content.

  The following aspects are obtained by the present invention. Each aspect is divided into sections, each section is numbered, and is described in the form of quoting the section numbers of other sections as necessary. In this way, each section is referred to the section number of the other section. By describing in the format, the technical features described in each section can be appropriately made independent according to the properties.

(1) An image is displayed to an observer by projecting image light onto an observation eye that is one eye of the observer in a state of being mounted on the head of the observer, thereby at least one of the objects in the target space A head mounted display that allows an observer to view an object in association with at least one reference content,
A determination unit for determining whether the observation eye is the left eye or the right eye of the observer;
Based on the determination result of the determination unit, on the image display surface of the head mounted display, the at least one reference content is observed from the image of the at least one object that the observer observes among the eyes of the observer. And a display control unit for displaying at a position deviated in a first direction away from a non-observing eye that is not an eye.

(2) The at least one object includes a plurality of objects existing in an observer's field of view determined relative to the position and orientation of the observation eye,
The head mounted display further includes
Based on at least one of the information related to the posture of the observer and the information related to the positions of the plurality of objects, any one of the plurality of objects is regarded as the attention object that the observer is substantially most interested in. A target object estimation unit to be estimated;
A reference content selection unit that selects a reference content corresponding to the estimated object of interest from among a plurality of reference content that can be displayed by the head mounted display;
First display position determination for determining the display position of the selected reference content based on the detection result of the position of the observation eye to a position deviating from the image of the object of interest observed by the observer in the first direction. And
When the reference content is displayed at the determined display position, there is a possibility that the reference content is displayed overlapping with an image of at least one non-attention object excluding the attention object among the plurality of objects. In some cases, a second display position determination is made to change the display position from the image of the at least one non-attention object to a position deviating in a second direction predetermined so as to intersect the first direction. The head mounted display according to item (1), including:

(3) The at least one object includes a plurality of objects existing in a field of view determined relatively to the position and orientation of the observation eye,
The head mounted display further includes
A first display size determining unit that determines a display size of the reference content to a predetermined first size;
When the reference content is displayed in the first size, there is a possibility that a part of the reference content may not be displayed in the image display area visually recognized by the observer during use of the head mounted display. The head mounted display according to (1) or (2), further including: a second display size determining unit that changes a display size of the second display size to a predetermined second size so as to be smaller than the first size.

(4) An image is displayed to the observer by projecting image light onto an observation eye that is one eye of the observer in a state of being mounted on the head of the observer, whereby at least one in the target space A method of operating a head-mounted display that allows an observer to view an object and at least one reference content associated with each other, comprising:
A determination step of determining whether the observation eye is the left eye or the right eye of the observer;
Based on the determination result in the determination step, on the image display surface of the head-mounted display, the at least one reference content is extracted from the image of the at least one object that the observer observes. A display control step of displaying at a position deviating in the first direction away from the non-observing eye that is not the observing eye.

(5) The at least one object includes a plurality of objects existing in a field of view determined relatively to the position and orientation of the observation eye,
The method further includes:
A target object estimation step of estimating any one of the plurality of objects as a target object that the observer is substantially most interested in;
A reference content selection step of selecting a reference content corresponding to the estimated object of interest from among a plurality of reference contents that can be displayed by the head mounted display;
First display position determination for determining the display position of the selected reference content based on the detection result of the position of the observation eye to a position deviating from the image of the object of interest observed by the observer in the first direction. Process,
When the reference content is displayed at the determined display position, there is a possibility that the reference content is displayed overlapping with an image of at least one non-attention object excluding the attention object among the plurality of objects. In some cases, a second display position determination is made to change the display position from the image of the at least one non-attention object to a position deviating in a second direction predetermined so as to intersect the first direction. The method according to item (4), comprising the steps of:

(6) A program executed by a computer to execute the method according to (4) or (5).

(7) Display configured to be able to be emitted by superimposing image light indicating reference content related to an object existing in the real external environment and external light indicating an image of the real external environment, and capable of being mounted on an observer's head Unit,
A position acquisition unit capable of acquiring the position of the display unit;
An orientation acquisition unit capable of acquiring the orientation of the display unit;
A mounting position acquisition unit capable of acquiring a mounting position of the display unit when the display unit is mounted by an observer;
A control unit for controlling the operation of the display unit,
Its control part is
Content acquisition means capable of acquiring the reference content related to the object based on the acquired position and orientation of the display unit;
Based on the acquisition result of the mounting position acquisition unit, a determination unit that determines on which side of the eyes of the observer the display unit is mounted;
Determining means for determining a display position of the acquired reference content;
The determination means is positionally associated with the object observation position at which the object related to the acquired reference content is observed, and shifted from the object observation position toward the eye determined by the determination means A head-mounted display that determines a position as a display position of the reference content.

  According to the present invention, since the reference content is displayed at a position away from the object image in a direction away from the non-observing eye, the observer's line of sight when the observation eye observes the reference content is on the non-observing eye side. Therefore, the visibility when the viewer visually recognizes the reference content is improved.

FIG. 1 is a perspective view showing a head-mounted display (HMD) according to an exemplary embodiment of the present invention in a left-eye observation mode and with a frame used for mounting on an observer's head. FIG. 2 is a longitudinal sectional view showing the display unit shown in FIG. FIG. 3 shows a comparison between an image visually recognized by the observer when the HMD shown in FIG. 1 is operated in the left eye observation mode and an image visually recognized by the observer when operated in the right eye observation mode. FIG. FIG. 4 is a functional block diagram conceptually showing the configuration of the HMD shown in FIG. FIG. 5A is a conceptual diagram showing the content related information stored in the content information storage unit shown in FIG. 4 in a tabular form, and FIG. 5B is the content related information shown in FIG. It is a figure for demonstrating the definition of. FIG. 6 is a bird's-eye view of the entire exhibition room from the top in order to explain how an observer is viewing a plurality of exhibits in a certain exhibition room while using the HMD shown in FIG. It is a figure shown by doing. FIG. 7A is a diagram for explaining how the display position of the reference content on the image display surface of the HMD shown in FIG. 1 is determined relative to the position of the image of the object of interest. FIG. 7B is a diagram for specifically explaining the situation. FIG. 8 is a flowchart conceptually showing a part of the image display method executed when the image display program stored in the memory shown in FIG. 4 is executed by the processor shown in FIG. FIG. 9 is a flowchart conceptually showing another part of the image display method.

  Hereinafter, some more specific and exemplary embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a perspective view of a head mounted display (hereinafter abbreviated as “HMD”) 10 according to an exemplary embodiment of the present invention, and FIG.

  First, to explain schematically, the HMD 10 is used by being mounted on the head of an observer so as to be positioned in front of the eyes of the observer. In the state of use, the HMD 10 guides external light representing an image of the real external environment (an example of the “target space”) positioned in front of the observer to the observer's eyes. Observe objects that exist in the outside world (for example, three-dimensional objects such as humans, machines, buildings, animals, plants, sculptures, and manufactured products, and two-dimensional objects such as photographs, paintings, movies, and printed materials) Designed to be possible.

  Further, the HMD 10 generates reference content related to the object that the observer is observing (for example, an image that represents information to be provided to the observer, text, graphics, an annotation, etc.) according to the image signal. The generated reference content is designed to be displayed to the observer in association with the observation position of the object currently being observed. In one example, the object is an exhibit that is displayed in a display room in a facility such as a museum, and the reference content is text or graphics to supplement the exhibit. Annotation.

  Further, the HMD 10 enables an observer to observe an image using either the left eye or the right eye of the observer as an observation eye. The HMD 10 is a monocular type. In the HMD 10, a left eye observation mode (shown in FIG. 1) in which the left eye of the observer is an observation eye and a right eye observation mode (not shown) in which the right eye of the observer is an observation eye are selected according to the observer's selection. And can be switched.

  Further, as shown in FIG. 3, the HMD 10 displays the reference content on the image display surface of the HMD 10 at a position deviated from an object image (hereinafter referred to as “object image”) observed by the observer. To do. Specifically, the HMD 10 moves the reference content from the object image to the left eye of the observer based on the position and size of the object being observed by the observer on the image display surface of the HMD 10. A position deviating in a direction selected according to the rules described later (e.g., right next to the object observation position) from the direction to the left (usually left) and the direction from the left eye to the right eye (usually right) Display a position that is out of position or a position that is obliquely off.).

  Further, the HMD 10 detects whether the observation eye is the left eye or the right eye based on a signal from an appropriate sensor in the use state, and as shown in FIG. 3, the observation eye is the left eye. In a certain left-eye observation mode, the reference content L is displayed in a direction away from the right eye from the image of the object A, for example, in a left position away from the left direction, while a right-eye observation in which the observation eye is the right eye In the mode, the reference content R is displayed in a direction away from the left eye from the image of the object A, for example, in a right position deviated in the right direction. This is a rule for selecting the display position of the reference content from the left position and the right position.

  Next, the configuration of the optical system employed by the HMD 10 will be described with reference to FIGS.

  As shown in FIG. 1, the HMD 10 includes a display unit 12, an attachment 14, and a control unit 16. The control unit 16 is connected to the display unit 12 via a cable 18 (including at least a signal line, usually including a power line) in a wired manner (or a wireless manner is possible), so that data between the two can be obtained. And signals can be exchanged.

  The display unit 12 is a monocular type, and is configured to project image light representing an image onto an observation eye that is one of the eyes of the observer (the other eye is a non-observation eye). The display unit 12 is an optical see-through type, that is, by causing image light representing an image corresponding to an image signal and external light from the real external environment located in front of the observer to be superimposed on the observation eye, A method is employed in which an observer can observe an image corresponding to an image signal superimposed on an image of the actual outside world. However, in the display unit 12, the present invention can be implemented in a mode in which the optical see-through type is changed to the video see-through type described above.

  As shown in FIG. 1, a frame 20 as a member to be attached is attached to the observer's head, an attachment 14 is attached to the frame 20, and a display unit 12 is attached to the attachment 14. As a result, the display unit 12 is attached to the frame 20 via the attachment 14.

  In the example shown in FIG. 1, the display unit 12 is attached to the left side portion 30 </ b> L of the frame 20, whereby the HMD 10 is used in the left eye observation mode, but the display unit 12 is attached to the right side of the frame 20 by the attachment 14. It can be used by changing between the portion 30R and the left portion 30L.

  Specifically, the attachment 14 includes a left-eye frame side member 40L fixed to the left side 30L of the frame 20, and a right-eye frame side member 40R (not shown) fixed to the right side 30R of the frame 20. It has. The attachment 14 has a display unit attached to the left-eye frame side member 40L attached to the left side 30L of the frame 20 in order to realize a left-eye observation mode in which the observer's left eye is an observation eye according to the observer's selection. 12 and the right-eye frame side member attached to the right side 30R of the frame 20 in order to realize the right eye observation mode in which the right eye of the observer is the observation eye (see FIG. 1). The state is switched to a state in which the display unit 12 is attached to 40R (not shown).

  The display unit 12 is used with its direction reversed between the right eye observation mode and the left eye observation mode. Specifically, the direction in which the display unit 12 is mounted on the frame 20 via the attachment 14 is set between the right-eye observation mode and the left-eye observation mode with respect to the left-right direction of the observer in the head-mounted state. They are reversed with respect to each other within a substantially parallel vertical plane (a plurality of planes extending in the vertical direction and parallel to the horizontal direction).

  As shown in FIG. 1, the display unit 12 has a housing 200 that is generally hollow extending in the longitudinal direction thereof. The housing 200 is made of synthetic resin. Housed in the housing 200 is an image light forming unit 202 (see FIG. 2) that forms image light for displaying reference content as a display image.

  As shown in FIG. 2, the display unit 12 includes a main body portion 210 that accommodates the image light forming unit 202 and an emission port 212 that emits image light formed by the image light forming unit 202 toward the observation eye. The emission port portions 214 are arranged so as to be aligned with each other along the longitudinal direction of the display unit 12. The display unit 12 further includes a connecting mechanism 270 that removably connects the main body portion 210 and the emission port portion 214 to each other.

  As shown in FIG. 2, the image light forming unit 202 generates the image light by two-dimensional spatially modulating incident light, and an LCD 220 (an example of a display unit) as a generally plate-shaped spatial light modulation element. Have The LCD 220 is a liquid crystal display in which a plurality of pixels are arranged two-dimensionally.

  In addition, in the present embodiment, the display unit 12 is a spatial light modulation type. However, the retinal scanning type, that is, a light beam from a light source such as a laser is scanned by a scanner, and the scanned light beam is observed by an observer. You may change to what is projected on the retina.

  The image light forming unit 202 further includes a plate-like driver 222 (an example of a display control unit). The driver 222 is electrically connected to the LCD 220 via a cable (not shown). The driver 222 drives the LCD 220 based on an image signal input from the outside, thereby causing the LCD 220 to emit image light representing an image to be displayed to the observer. The LCD 220 includes a backlight light source, but a light source independent of the LCD 220 may be used instead of the light source.

  As shown in FIG. 2, the display unit 12 further includes an eyepiece optical system 246. The eyepiece optical system 246 is configured as a one-dimensional array in which a plurality of lenses as a plurality of optical elements are arranged in series. These lenses share the same optical axis. The optical axis extends in a straight line parallel to the longitudinal direction of the housing 200 and without being bent by a component such as a mirror. The eyepiece optical system 246 is accommodated in a portion of the housing 200 that belongs to the main body portion 210.

  The eyepiece optical system 246 includes a terminal lens 248 located on the most downstream side among a plurality of lenses belonging to the eyepiece optical system 246, and a protective transparent body (for example, a transparent disc made of synthetic resin) 250 disposed downstream of the terminal lens 248. And.

  The exit portion 214 is a half mirror as an example of a partial reflection / partial transmission optical element that bends the image light emitted from the eyepiece optical system 246 and guides it to the observation eye (an example of a deflection member that deflects the image light toward the observation eye). 260.

  The image light emitted from the eyepiece optical system 246 is reflected by the half mirror 260, passes through the pupil of the observation eye, and enters the retina (not shown). Thereby, the observer can observe the two-dimensional image as a virtual image. Not only the image light reflected by the half mirror 260 but also external light, which is light from the actual external environment, passes through the half mirror 260 and enters the observation eye. As a result, the observer can observe the actual outside world in parallel with the observation of the image displayed by the image light.

  Next, with reference to FIG. 4, a hardware configuration of a part of the HMD 10 that displays the reference content in association with the object will be described.

  The display unit 12 includes an input device 12a and an output device 12b.

  The input device 12a is the position of the observer wearing the display unit 12 (for example, the position of the head of the observer, the position of the display unit 12, the position on the global coordinate system, and the facility A position detection unit 300 that detects a position on a fixed local coordinate system). The position detection unit 300 is, for example, a device mounted on the display unit 12 for detecting the position of the observer by the global positioning system GPS, or for detecting the position of the observer by the three-point positioning method. In addition, a device mounted on the display unit 12 can be used.

  The input device 12a further includes the orientation of the observer (for example, the orientation of the observer's head, the orientation of the display unit 12, and the orientation on the global coordinate system or the local coordinate system fixed to the facility. A direction detecting unit 302 for detecting the direction of The orientation detection unit 302 is, for example, a device mounted on the display unit 12 for detecting the orientation of the observer using an electronic compass, or an angular acceleration sensor or an angular velocity sensor mounted on the display unit 12 as a main component. A device mounted on the display unit 12 can be used to calculate the angle of the observer by time-integrating the detected value.

  As another example of the direction detection unit 302, a configuration that detects a relative direction with respect to an object may be employed. In this case, for example, a camera is connected to the display unit 12, and the relative direction of the display unit 12 with respect to the object may be detected based on an image obtained by imaging the object with the camera. . In this case, if necessary, the detection result of the position detection unit 300 may be used to specify the position information of the object to be imaged.

  The input device 12a further includes a mounting detection unit 304 that detects a state in which the display unit 12 is mounted on the head of the observer, that is, the mounting of the display unit 12. The attachment detection unit 304 can employ, for example, a method of detecting attachment of the display unit 12 by a contact type or a non-contact type. When the non-contact type is adopted, the wearing detection unit 304 has, for example, a light emitting unit and a light receiving unit in a posture facing the surface of the observer's head, and electromagnetic waves (for example, red light) emitted from the light emitting unit. The external light is a device that is mounted on the display unit 12 in order to detect the mounting of the display unit 12 when the light receiving unit receives a portion reflected from the head of the observer.

  The input device 12a further determines whether the display unit 12 is mounted on the right eye side or the left eye side of the observer in a state where the display unit 12 is mounted on the observer's head. And a mounting position detection unit 306 that detects whether the observation eye is the right eye or the left eye. The mounting position detection unit 306 can employ, for example, a method of detecting the mounting position of the display unit 12 by a contact type or a non-contact type. The mounting position detection unit 306 can employ, for example, a method of detecting the mounting position of the display unit 12 relative to the observer's head or the frame 20. In this case, for example, the mounting position detection unit 306 determines which of the left eye frame side member 40L and the right eye frame side member 40R constituting the attachment 14 the display unit 12 is mounted on. It is possible to detect by contact type or non-contact type. The mounting position detection unit 306 may adopt a configuration that detects whether the mounting position of the display unit 12 is the right eye side or the left eye side by receiving an input from an observer. .

  The mounting position detection unit 306 can employ, for example, a method of detecting the mounting position of the display unit 12 relative to the absolute space or the surrounding local fixed space. In this case, for example, the mounting position detection unit 306 can be configured with a gravity sensor as a main component or a distance measurement sensor as a main component.

  On the other hand, the output device 12b has the above-described driver 222 and LCD 220 for image display.

  As shown in FIG. 4, the control unit 16 includes a processor 310 and a memory 312. The memory 312 is an example of a recording medium, and a program storage unit in which an image display program (see FIGS. 8 and 9) described later is recorded in advance, and reference content is displayed by the HMD 10 in association with the position of the object. And a content information storage unit for storing information necessary for the storage. The content information storage unit includes, for example, a content information storage unit that stores content related information related to a plurality of reference contents that can be displayed by the HMD 10 in association with a plurality of corresponding objects.

  Note that one reference content is not necessarily displayed in association with one object, and a plurality of reference contents may be displayed in association with one object.

  In one example in FIG. 5A, the content-related information includes information on a plurality of reference contents that can be displayed by the HMD 10, and for each reference content, data on an image representing each reference content (for example, the image) And a horizontal size H and a vertical size V of the image on the image display surface.

  In this example, the content related information further includes object related information related to the object, that is, exhibit related information related to the exhibit. Specifically, the exhibition related information includes information on a plurality of exhibits existing in the real world, and for each exhibit, data representing the name of the exhibit and the coordinates of the representative point of the exhibit. It includes data representing the position and size-related information for specifying the size of the exhibit (not the size of the image of the exhibit on the image display surface, but the size of the actual exhibit). .

  An example of the size-related information is a point on the exhibit, which is a position coordinate of another point referred to together with the representative point (hereinafter referred to as “reference point”). For example, it is possible to use information that directly represents the size of an exhibit (for example, maximum length, diameter, radius, etc.).

  FIG. 5B shows an observation when the observer observes the exhibit A in the example shown in FIG. 5A and the reference content A to be displayed in association with it through the HMD 10 at the same time. An example of the whole image that the person visually recognizes on the image display surface, that is, in the image display area is shown.

  In the example shown in FIG. 5B, the exhibit A is not an image that faithfully represents the real object, but is a solid that geometrically approximates the actual object and contains the actual object, and is predetermined. It is expressed as an image representing a figure (hereinafter, referred to as “minimum approximate solid”) having a minimum volume (hereinafter referred to as “minimum approximate solid”). Further, in this example, the minimum approximate solid is a polyhedron having a plurality of vertices, a certain point on the minimum approximate solid is selected as a representative point of the exhibit A, and the reference point of the exhibit A As described above, a point that is substantially farthest from the representative point among a plurality of points on the minimum approximate solid is selected.

  Specifically, in the example shown in FIG. 5B, a hexahedron having eight vertices and six planes is selected as such a minimum approximate solid, and a hexahedron is configured as the representative point. Any one of the eight vertices is selected, and, as the reference point, a vertex that is diagonal to the representative point is selected from the eight vertices.

  Another example of such a minimum approximate solid is a sphere that approximates exhibit A, as shown in FIG. In this example, the center point of the sphere is selected as the representative point of the sphere, and the radius of the sphere is selected as the size of the sphere.

  As shown in FIG. 4, the control unit 16 further includes a communication module 320, and the communication module 320 is connected to an external server 322 by wireless or wired communication. In response to the request from the control unit 16, the server 322 transmits the content related information to the control unit 16. When the processor 310 receives the content-related information, the processor 310 temporarily stores it in the memory 312.

  In the present embodiment, the content-related information is temporarily received from the server 322 and stored in the HMD 10 itself. For example, the HMD 10 does not receive from the server 322 and stores the memory in the server 322. Necessary data processing can be performed using (not shown).

  The processor 310 implements a plurality of functions by executing the image display program while using data stored in the memory 312.

  Specifically, the processor 310 includes an object acquisition unit 400, a reference content display unit 402, and an image signal generation unit 404. The image signal generation unit 404 generates an image signal based on the data from the reference content display unit 402 and supplies the image signal to the LCD 230 via the driver 222. Thereby, the reference content is displayed to the observer.

  The object acquisition unit 400 receives the content related information from the server 322 via the communication module 320. The content related information includes the object related information. The object acquisition unit 400 further extracts a plurality of objects existing in the observer's field of view from the plurality of objects represented by the object related information. Note that although there may be no single object or only one object in the observer's field of view, it is assumed for the sake of convenience that there are a plurality of objects.

  The function of the object acquisition unit 400 will be described more specifically. The object acquisition unit 400 selects a plurality of objects in the observer's field of view from a plurality of objects existing in the actual outside world. Here, the “viewer's field of view” is a three-dimensional angular range determined relatively to the position (x0, y0, z0) and direction (z-axis direction) of the observation eye, as shown in FIG. Thus, the range is generally fan-shaped in plan view. Here, the “z-axis direction” means the front direction of the observer, and this direction does not necessarily coincide with the line-of-sight direction of the observer. The fan-shaped range is a range that is referred to in order to detect an object to be displayed in association with the reference content from among a plurality of objects existing in the actual outside world. Focusing on this, this range is hereinafter also referred to as “object detection range”.

  FIG. 6 shows a state in which an observer is viewing a plurality of exhibits (for example, dinosaur replicas) in a certain exhibition room while using the HMD 10, and the whole exhibition room is viewed from directly above. This is shown in the figure shown. In this exhibition room, a plurality of antennas are installed in advance, and the HMD 10 has three antennas that are closest to the HMD 10 among the antennas via at least the position detection unit 300 among the position detection unit 300 and the orientation detection unit 302. By utilizing the principle of triangulation using an antenna, the position (x0, y0, z0) of the observer is detected on a local coordinate system or a global coordinate system fixed in the exhibition room.

  In an example using the principle of triangulation, a signal is transmitted from each of the plurality of antennas simultaneously so that each antenna is directed to a plurality of other antennas, and the signals are transmitted to the plurality of antennas. The position of the observer is detected based on the difference or ratio of reception timing. In another example using the principle of triangulation, a signal having the same intensity is transmitted from each of the plurality of antennas simultaneously so as to go from each antenna to the plurality of other antennas. The position of the observer is detected based on the difference or ratio of the received intensity of the signals when the signals are received by the plurality of antennas, respectively.

  As illustrated in FIG. 4, the reference content display unit 402 further includes an attention object estimation unit 410 and a reference content selection unit 412.

  The attention object estimation unit 410 estimates any one of the plurality of objects existing within the object detection range as the attention object that the observer is substantially most interested in. In one example, the object closest to the observer is estimated as the object of interest. In another example, an observer's line of sight (which is an example of an observer's posture) is detected, and an object that intersects the detected line of sight among the plurality of objects is estimated as a target object. The reference content selection unit 412 performs the estimation from a plurality of reference contents defined by the content-related information (that is, a plurality of reference contents that can be displayed by the HMD 10 by referring to the content-related information). The reference content corresponding to the noted object of interest is selected.

  The reference content display unit 402 further includes a first display position determination unit 414, a second display position determination unit 416, and a third display position determination unit 418.

  As illustrated in FIG. 7A, the first display position determination unit 414 displays the first candidate display position RC1 of the reference content (represented by a horizontally long rectangle) from the image (represented by a circle) of the object of interest A. Then, the position is determined to be out of the left direction and the right direction by the movement amount α in the direction selected in accordance with the position of the observation eye (the left direction in the illustrated example).

  If the second display position determination unit 416 displays the reference content at the determined first candidate display position RC1, the second display position determination unit 416 may be displayed in an overlapping manner with an image (represented by a circle) of another object B. If there is, the first candidate display position RC1 of the reference content is moved by a movement amount β in one of the upper direction and the lower direction (downward in the illustrated example), thereby A second candidate display position RC2 for the reference content is determined.

  If the third display position determining unit 418 displays the reference content at the determined second candidate display position RC2, it may be displayed overlapping with another object C image (represented by a circle). If the second candidate display position RC2 of the reference content is in the direction opposite to the direction selected according to the position of the observation eye, the second candidate display position RC2 of the reference content (in the illustrated example, the right direction) ) With the movement amount γ, thereby determining the final display position RC3 of the reference content.

  The reference content display unit 402 further includes a first display size determination unit 420 and a second display size determination unit 422.

  The first display size determination unit 420 determines the display size of the reference content to be a predetermined first size. When the reference content is displayed in the first size, the second display size determining unit 422 determines the display size of the reference content when there is a possibility that a part of the reference content is not displayed in the image display area. The size is changed to a predetermined second size so as to be smaller than the size.

  In an example of the method of changing the display size of the reference content from the first size to the second size, when there are a plurality of information items included in the reference content to be displayed, a part of the information items is deleted The reference content is displayed. In another example, the reference content to be displayed is displayed in a state reduced at a predetermined magnification.

  Next, the image display method and the software configuration for executing the image display method, that is, the image display program will be described with reference to FIGS.

  FIG. 8 conceptually shows a flowchart of an image display method executed by the processor 310 executing the image display program. The image display program is executed by the processor 310 when the power of the HMD 10 is turned on.

  When this image display program is executed, first, in step S1, it is awaited that the viewer detects the mounting of the display unit 12 on the viewer's head based on the signal from the mounting detector 304. . If the mounting of the display unit 12 is detected, it is determined in step S2 based on the signal from the mounting position detection unit 306 whether the mounting position of the display unit 12 is on the right eye side of the observer. If the mounting position is on the right eye side, in step S3, it is determined that the current position is in the right eye observation mode. On the other hand, when the mounting position of the display unit is on the left eye side of the observer, in step S4, it is determined that the present time is in the left eye observation mode.

  In either case, thereafter, in step S5, the position and orientation of the observer are detected based on the signals from the position detection unit 300 and the orientation detection unit 302. Next, in step S6, based on the detected position and orientation, and information for geometrically specifying the observer's view range (stored in the memory 312), the object detection range (FIG. 8 is simply expressed as “detection range”).

  Subsequently, in step S 7, in response to a request from the observer, the content related information and the object related information are received from the server 322, and the received information is stored in the memory 312. Thereafter, in step S8, based on the relationship between the calculated object detection range and the object related information (including the three-dimensional position coordinates of each object), a plurality of objects existing in the object detection range are detected. .

  Subsequently, in step S9, based on the object related information and the detected value of the observer's position, the object located at the position closest to the observer among the plurality of objects existing in the object detection range is the object of interest. Is estimated as In the example shown in FIG. 6, the exhibits B and C exist within the object detection range, but since the exhibit B is closer to the observer than the exhibit C, the exhibit B is the object of interest. Selected as.

  After that, in step S10 shown in FIG. 8, among the plurality of reference contents represented by the content related information, the one corresponding to the estimated attention object is the reference content to be displayed in association with the attention object this time. Selected. Subsequently, in step S11, the display size of the current reference content is determined to be a predetermined first size.

  Thereafter, in step S12, the display unit 12 and the object of interest (positional coordinates (position coordinates of the representative point)) based on the detected values of the observer's position and orientation and the portion related to the object of interest (position coordinates of the representative point) of the object related information. Relative position in the three-dimensional direction between the real object and the position where the target object appears on the image display surface when the observer observes the target object is fixed on the image display surface. Defined using local coordinate system.

  Subsequently, in step S13, based on the detected value of the position of the observer and the portion related to the object of interest in the object related information (position coordinates of the representative point and data representing the size of the object) By focusing on the similarity between two similar triangles having a common vertex at the position of the person's retina, and a small triangle having the image display surface as the base and a large triangle having the target object (the real object) as the base The size of the image of the object of interest (hereinafter simply referred to as “image of the object of interest”) when the observer observes the object of interest on the image display surface is calculated.

  Thereafter, in step S14, the display position of the reference content on the image display surface (for example, the position of the representative point preset in the reference content or the position of the geometric center point of the reference content) It is calculated as a position deviated from the image in a direction away from the non-observing eye.

  In the example shown in FIG. 7A, as shown in FIG. 7B, when this step S14 is executed, the display position Ycon in the vertical direction of the reference content RC becomes the vertical direction of the image of the object of interest A. A horizontal position Xobj of the image of the object of interest on the image display surface, which is determined to be substantially the same as the position Jobj, and the display position Xcon of the reference content in the horizontal direction is calculated as described above, The image size X1 of the object of interest on the image display surface (which may be ignored) calculated as described above, the display area of the reference content and the image of the object of interest on the image display surface, The minimum gap length X2 (stored as a fixed value in the memory 312) and the reference on the image display surface Since (also if it is ignored) and the display size X3 of content is calculated as the image of the reference content on the image display surface is formed so as not to overlap any and image object of interest.

In one example, as shown in FIG. 7B, the display position Xcon in the horizontal direction of the reference content is
Xcon = Xobj-X1-X2-X3
It is calculated using the following formula.

  Thereafter, in step S15 shown in FIG. 9, the virtual display state in which the reference content is displayed together with the plurality of objects without actually displaying the reference content at the calculated display position is, for example, bit Calculated using the map memory method. Subsequently, in step S16, it is determined whether or not another object (for example, object B in the example shown in FIG. 7) exists behind the reference content in the virtual display state.

  If there is no other object behind the reference content, the process immediately proceeds to step S20, but if it exists, the display position of the reference content is set to an image of another object as much as possible by executing steps S17 to S19. It is modified so that the reference content is displayed without overlapping.

  Specifically, in step S17, the display position of the reference content (calculated by executing step S14) is moved downward by a movement amount β (stored as a fixed value in the memory 312). Be made. Subsequently, in step S18, whether or not there is another object (for example, object C in the example shown in FIG. 7) behind the reference content in the virtual display state updated as described above. Determined.

  If there is no other object behind the reference content, the process immediately proceeds to step S20, but if it exists, in step S19, the display position of the reference content is moved toward the non-observing eye. It is moved by an amount γ (stored as a fixed value in the memory 312). Thereafter, the process proceeds to step S20.

  In any case, in step S20, it is determined whether or not the reference content is in the image display area in the virtual display state. If the reference content is not in the image display area, it is prohibited to actually display the reference content in step S21. Thereafter, the process proceeds to step S25.

  On the other hand, if the reference content is in the image display area in the virtual display state, in step S22, the entire reference content is present in the image display area in the virtual display state. It is determined whether or not to do so. If the entire reference content exists in the image display area, image data representing the reference content without change is created in step S23, and the reference content is actually displayed based on the image data. .

  On the other hand, when the entire reference content does not exist in the image display area, that is, when the reference content is displayed in a partially missing state, the reference content is edited in step S24. Image data is created in a state where the display size is reduced (for example, as described above, the display size is reduced by reducing the number of information items, or the display size is reduced without changing the number of information items), Based on the image data, the reference content is actually displayed. Thereafter, the process proceeds to step S25.

  In any case, after that, in step S25, based on the signal from the attachment detection unit 304, it is determined whether or not the movement of the observer removing the display unit 12 from the observer head is detected. When the removal is detected, the execution of the image display program ends. When the removal is not detected, the process returns to step S5.

  As is clear from the above description, according to the present embodiment, in the HMD 10 adopting the monocular type and the optical see-through type, the reference content related to the target object observed by the observer becomes the image of the target object. It is displayed in a position that is associated with the position and deviated from the image of the object of interest away from the non-observed eye. For example, in the left eye observation mode shown in FIG. 1, the reference content is displayed at a position away from the image of the object of interest in a direction away from the right eye, which is the observer's non-observation eye.

  Therefore, according to the present embodiment, for example, in the left eye observation mode, the reference content is different from the case where the reference content is displayed at a position deviating from the image of the object of interest so as to approach the right eye that is the non-observation eye. The line of sight of the observer when the observing eye observes the reference content is greatly inclined to the right eye side. As a result, the line of sight is the right side of the observer in the housing 200, particularly, at both ends of the housing 200. The possibility of intersecting with the near part or the half mirror 260 mounted on the display unit 12 at a position near the right eye of the observer is reduced.

  Regardless of the left-eye observation mode or the right-eye observation mode, when the reference content is displayed at a position away from the image of the target content in a direction away from the right eye, the left-eye observation mode is used. The viewer can observe the reference content with high visibility, but the right-eye observation mode causes a left-right difference in visibility so that the viewer cannot observe the reference content with high visibility. End up.

  On the other hand, according to the present embodiment, the relative position of the reference content with respect to the image of the content of interest is switched geometrically depending on whether the left-eye observation mode or the right-eye observation mode. The observer can observe the reference content with high visibility regardless of the left-eye observation mode or the right-eye observation mode.

  Further, according to the present embodiment, when there are a plurality of objects in the field of view of the observer, information regarding the posture of the observer (including eye movement) and the relative position of these objects with respect to the display unit 12 are described. Based on the information or the like, one of these objects is extracted as a target object, and the reference content is displayed only in relation to the target object.

  Therefore, according to the present embodiment, even when there are a plurality of objects in the field of view of the observer, only one reference content is displayed, and thus there are a plurality of objects in the recording image display area. Reference content that is estimated to be really necessary for the viewer is displayed in the narrow empty space even if the empty space existing in the image display area is narrow because the images are close to each other. Easy to do.

  Further, according to the present embodiment, when there are a plurality of objects in the image display area, the reference content is displayed so that the reference content corresponding to the object of interest does not overlap with any object as much as possible. The position is determined. The display position of the reference content is not determined so as not to overlap with the image of the target object related to the reference content, and is not disturbed by the target object image. Of course, the position of the image of another object existing around the reference content is also considered, and the display position of the reference content is determined so as not to overlap with the image of any object. Thus, the observer is not disturbed by the image of the object of interest associated with the reference content, nor is it obstructed by the image of other objects existing around the reference content. It becomes possible to observe, and this also improves the visibility of the reference content.

  As is apparent from the above description, in this embodiment, for the sake of convenience of explanation, steps S1 to S4 in FIG. 8 constitute an example of the detection step, and steps S5 to S9 in FIG. Step S10 in the figure constitutes an example of the reference content selection step, and steps S12 to S14 in the figure constitute an example of the first display position determination step. Further, it can be considered that steps S15 to S17 in FIG. 9 constitute an example of the second display position determining step.

  As mentioned above, although several embodiment of this invention was described in detail based on drawing, these are illustrations and are based on the knowledge of those skilled in the art including the aspect as described in the above-mentioned [summary of invention] column. The present invention can be implemented in other forms with various modifications and improvements.

   For example, in the embodiment shown in FIG. 9, after the display position of the reference content is automatically determined so as to be on the right eye side or the left eye side with respect to the position of the target object, steps S17 to S17 are performed. By executing S19, the determined display position (the first automatically determined display position) is automatically performed without the intervention of the observer as necessary so that the reference content does not overlap with another object. To be corrected.

  On the other hand, in another embodiment, the correction of the display position determined automatically first is performed manually. In one example, in the step corresponding to step S15, the reference content is actually displayed at the display position calculated by executing steps S12 to S14 shown in FIG. After that, in a step corresponding to step S16, when it is determined that another object (for example, object B in the example shown in FIG. 7) exists behind the reference content, by receiving an input from the observer, The display position of the reference content can be changed.

  In order to accept this input, for example, the viewer is prompted to change the display position of the reference content (for example, a visual, auditory or tactile instruction). When an input (for example, an operation on a physical operation unit (for example, a key) or a virtual operation unit (for example, an icon) (not shown) or voice input is performed), the display position of the reference content is based on the input. Is changed. In this way, when another object exists behind the reference content, the observer can freely select a place where the reference content is arranged.

Claims (7)

An image is displayed to the observer by projecting image light onto an observation eye that is one eye of the observer in a state of being attached to the head of the observer, thereby at least one object in the target space and at least A head-mounted display that allows an observer to view one reference content in relation to each other,
A determination unit for determining whether the observation eye is the left eye or the right eye of the observer;
Based on the determination result of the determination unit, on the image display surface of the head mounted display, the at least one reference content is observed from the image of the at least one object that the observer observes among the eyes of the observer. And a display control unit for displaying at a position deviated in a first direction away from a non-observing eye that is not an eye. The at least one object includes a plurality of objects existing in an observer's field of view determined relative to the position and orientation of the observation eye;
The head mounted display further includes
Based on at least one of the information related to the posture of the observer and the information related to the positions of the plurality of objects, any one of the plurality of objects is regarded as the attention object that the observer is substantially most interested in. A target object estimation unit to be estimated;
A reference content selection unit that selects a reference content corresponding to the estimated object of interest from among a plurality of reference content that can be displayed by the head mounted display;
First display position determination for determining the display position of the selected reference content based on the detection result of the position of the observation eye to a position deviating from the image of the object of interest observed by the observer in the first direction. And
When the reference content is displayed at the determined display position, there is a possibility that the reference content is displayed overlapping with an image of at least one non-attention object excluding the attention object among the plurality of objects. In some cases, a second display position determination is made to change the display position from the image of the at least one non-attention object to a position deviating in a second direction predetermined so as to intersect the first direction. The head mounted display according to claim 1, comprising: The at least one object includes a plurality of objects existing in a field of view that is determined relative to the position and orientation of the observation eye,
The head mounted display further includes
A first display size determining unit that determines a display size of the reference content to a predetermined first size;
When the reference content is displayed in the first size, there is a possibility that a part of the reference content may not be displayed in the image display area visually recognized by the observer during use of the head mounted display. The head-mounted display according to claim 1, further comprising: a second display size determining unit that changes a display size of the second display size to a predetermined second size so as to be smaller than the first size. An image is displayed to the observer by projecting image light onto an observation eye that is one eye of the observer in a state of being attached to the head of the observer, thereby at least one object in the target space and at least A method of operating a head mounted display that allows a viewer to view a reference content in association with each other, comprising:
A determination step of determining whether the observation eye is the left eye or the right eye of the observer;
Based on the determination result in the determination step, on the image display surface of the head-mounted display, the at least one reference content is extracted from the image of the at least one object that the observer observes. A display control step of displaying at a position deviating in the first direction away from the non-observing eye that is not the observing eye. The at least one object includes a plurality of objects existing in a field of view that is determined relative to the position and orientation of the observation eye,
The method further includes:
A target object estimation step of estimating any one of the plurality of objects as a target object that the observer is substantially most interested in;
A reference content selection step of selecting a reference content corresponding to the estimated object of interest from among a plurality of reference contents that can be displayed by the head mounted display;
First display position determination for determining the display position of the selected reference content based on the detection result of the position of the observation eye to a position deviating from the image of the object of interest observed by the observer in the first direction. Process,
When the reference content is displayed at the determined display position, there is a possibility that the reference content is displayed overlapping with an image of at least one non-attention object excluding the attention object among the plurality of objects. In some cases, a second display position determination is made to change the display position from the image of the at least one non-attention object to a position deviating in a second direction predetermined so as to intersect the first direction. The method according to claim 4, comprising the steps of:   A program executed by a computer to execute the method according to claim 4 or 5. A display unit configured to be able to emit by superimposing image light indicating reference content related to an object existing in the real external environment and external light indicating an image of the real external environment, and a display unit that can be mounted on the head of an observer;
A position acquisition unit capable of acquiring the position of the display unit;
An orientation acquisition unit capable of acquiring the orientation of the display unit;
A mounting position acquisition unit capable of acquiring a mounting position of the display unit when the display unit is mounted by an observer;
A control unit for controlling the operation of the display unit,
Its control part is
Content acquisition means capable of acquiring the reference content related to the object based on the acquired position and orientation of the display unit;
Based on the acquisition result of the mounting position acquisition unit, a determination unit that determines on which side of the eyes of the observer the display unit is mounted;
Determining means for determining a display position of the acquired reference content;
The determination means is positionally associated with the object observation position at which the object related to the acquired reference content is observed, and shifted from the object observation position toward the eye determined by the determination means A head-mounted display that determines a position as a display position of the reference content.

Images