JP2016536635A - System and method for reconfigurable projection augmented reality / virtual reality appliance - Google Patents

Abstract

A system comprising a head mounted display with eye tracking includes an attachment for reconstruction from a projection augmented reality application to a closed virtual reality application or mixed mode.

Description

  The present invention relates to the fields of virtual reality, augmented reality, board games, and video games. More particularly, the system is a global image combiner for multiple modes of operation in a reconfigurable head mounted display: projected image to surface, near to eye display, and graphic overlay. Enables near-to-eye display with

  Fixed optics head-mounted display headsets typically consist of one or more displays and relay optics that deliver computer-generated graphics to the user's eyes, and there are many examples. Additional fixed optics that combine light from the real world can be provided, allowing graphics to be overlaid on what the user sees in the real world. The display is often associated with a subsystem to track the user's line of sight to provide information driving the rendering of the CGI scene for stereoscopic viewing and to simulate 3D vision. .

FIG. 2 is a typical outward projection image headset comprising two projection display systems and two apertures for light returning from the surface to the user in the world, along with a camera for tracking markers. The figure of the wired connection system for the headset of FIG. FIG. 2 is a front view of the headset of FIG. 1 showing eye alignment with the projector. Figure 6 shows an alternative headset with anisotropic reflection. FIG. 4 is an alternative headset using one projector. FIG. 4 is an illustration of an active “marker” pad for use in eye tracking. FIG. 2 is a diagram of the optical path from and to the headset of FIG. FIG. 2 is a diagram of an optical path from a tracking marker light emitter to the headset of FIG. FIG. 6 is an optical path diagram for “clip-on” reconstruction into a closed virtual reality mode of operation. Diagram of hinged “flip-up” operation for switching modes. Front "perspective" view of the "clip-on" device in the closed position. Illustration of one-sided use of a “clip-on” device. FIG. 4 is an alternative “clip-on” reconstruction for mixed real / virtual mode. FIG. 6 is an alternative “clip-on” reconstruction with a camera for “electronic see-through” real / virtual mixed mode.

  The present invention relates to one or more displays having a mode of main operation, such as projection imaging, a line-of-sight tracking subsystem, attachments for relaying images directly to the user's eyes, and / or world image combing optics. And a headset comprising the system. The gaze tracking system provides the information necessary to render a stereoscopic view of a computer generated scene, such as, for example, those used for first person viewpoint based video games or simulations.

  The system of the present invention can be used for display or projection systems (FIGS. 1-5) and line-of-sight tracking systems (FIGS. 6-7), as well as mechanical changes in operating mode from projection to near-to-eye viewing. Eyeglasses (or a headset) equipped with a relay system (FIGS. 8 to 10) that can be attached to the camera.

  An embodiment of the glasses is shown in FIG. In FIG. 1, the frame 101 supports a pair of image projectors 102 and 104, one or more tracking cameras 103, and viewing lenses 105 and 106. A compartment 107 is shown that can hold batteries and driver electronics and wireless electronic communication devices. Instead, FIG. 2 shows an embodiment having a wired connection 201 to the circuit box 202. The circuit box 202 may include a connection unit 203 for both computer / cell phone interfaces such as HDMI (registered trademark) and / or a connection unit 204 for other peripheral devices such as USB. The circuit box 202 may also include a battery.

  The viewing lenses 105, 106 of FIG. 1, along with the projectors 102, 104, provide a means for excluding light emanating from the projector on the opposite side of the frame. This means can be selective orthogonal polarization (planer or circular), time-division multiplexed active shutter, spectral filtering (by emitter physics or software selected color or passive filtering), or in the art It can be by such other means known.

  As shown in FIG. 7a showing the projected augmented reality mode, the system uses a retroreflective material 701 to overlap most of the light 702 emitted by the projectors 102, 104 over the viewing lenses 105 and 106 via a path 703. Return to. Prior art (eg, Stanton US Pat. No. 6,535,182) teaches a system in which the projector is located on the side adjacent to the hinge of the frame. However, this makes the crosstalk of unwanted images from both sides low when the frame is large enough to be worn on the user's existing eyewear, while moving from the user's eyes to the projector. Due to the off-axis distance, there is a disadvantage that the brightness of the returned image is reduced. Prior art techniques (eg, Fisher US Pat. No. 5,572,229 and Ferguson US Pat. No. 5,606,458) have projected light into the user's line of sight. It also teaches the use of a beam splitter in front of the user's eye to be oriented coaxially, but this adds undesired forward weight and enlargement of the frame structure. FIG. 3 shows a preferred alignment of the embodiment of FIG. 1 where a beam splitter is not required and the projector is located directly above the center of each eye of the user. The projector may be mounted under the eyes, centered on each and the same centerline, and the retroreflective material is partially transparent so that the user can see the object placed behind it. It is noted that

  In an alternative embodiment, the alignment shown in FIG. 3 may be used with an anisotropic retroreflective screen so that the brightness pattern of the returned projected image decreases more quickly in the horizontal direction than in the vertical direction. Anisotropic retroreflectors are based on a slightly elliptical reflecting sphere aligned in the axis, or a holographic film on a specular surface, or other means known in the art of retroreflector manufacturing and autostereoscopic screens Can be manufactured. The spatial separation of left / right images according to this configuration is shown in FIG. In FIG. 4, the spectacle frame 401 is open and does not have a filtered viewing lens, but rather the anisotropic viewing return region 402 limits the light that intersects the opposite eye. The

An alternative embodiment using one projector is shown in FIG. In FIG. 5, projector 502 sequentially sends alternating frames and filtered viewing lenses 505, 506.
Selectively passes left and right images to the corresponding eyes. As described above, one projector 502 can either time-multiplex by switching the polarization orthogonality (when using either planar or circular polarization), or time multiplexing with an active shutter in the viewing lens, or view It is coordinated with the viewing lens by projecting on the left / right side the limited color that the spectral filter in the ining lens passes.

  In order to present a virtual form or an advanced form of augmented reality, it is necessary to calculate the user's line of sight. For the purposes of this specification, the line of sight originates from between the user's eyes and is a line that extends forward in parallel to the central projection lines of the projectors 102 and 104 that are mounted so as to be parallel to each other.

  The line-of-sight tracking subsystem consists of a headset camera or multiple cameras 103 attached to a line of central vision parallel to the central projection lines 102 and 104 and a “pad” as shown in FIG. It includes one “marker” or a plurality of markers that can take the form. In this embodiment, this pad or plate 601 comprises a set of five infrared light emitting diodes, in which the four outer units 602-605 are in constant output mode, while the offset inner diode 606 is an identification code pattern. Is modulated using. The power supply and modulation circuit for the emitter may be embedded in the material of the pad (not shown), and the emitter may be supplied by wire from other locations. The marker may also have a front surface with a retroreflective material that becomes part of the surface that returns the projected image to the headset. Multiple marker pads can be used with a given configuration having various codes broadcast by a modulated IR source, particularly as identified by headset firmware or software. An equivalent marker configuration will be apparent to designers in the art.

  FIG. 7 a shows a typical optical path from the projector on the headset to the retroreflective surface 701 that is attached to the frame 705. For retroreflective surfaces, by nature, the angle presented to the user is not critical, and the surface may have a fold, bend, or flat portion. FIG. 7b shows the optical path 705 of light emanating from the emitter marker pattern 704 that is tracked by the camera (103 in FIG. 1) to provide geometric data. This geometric data can be mathematically processed to calculate the user's line of sight relative to the fixed surface. In this figure, the marker of FIG. 6 is embedded in the surface 701 so that an opening is provided for IR emission. Alternatively, the surface may be transparent to IR and behind it may be a marker pad. For the purposes of this specification, the term “retroreflector” may be any surface from transparent to opaque that returns a substantial amount of the projected light directly to the projector.

The headset of FIG. 1 takes the enclosure type from the projection mode by a “clip on” optical relay system attachment that redirects the projector output to an imaging path that is directed directly to each of the user's corresponding eyes. Can be converted to a near-to-eye virtual reality display. A cross-sectional view of the optical path on one side of the attachment is shown in FIG. 8a. In this figure, the enclosure 801 is held at a fixed position by the clamping means on the projector housing 102 on the headset frame 101 by the hinge mechanism 805. Enclosure 801 holds in place the configuration of optical elements that direct the image generated by the projector to be collimated to produce a viewable image that is presented coaxially to the user's eyes. Means (not shown). In the illustrated embodiment, the image from projector 102 is oriented downward by mirror 802 and then forward by beam splitter 803 and then reflected by shaping mirror 804. The shaping mirror 804 provides a correctly polarized collimated image returning through the beam splitter 803 and the viewing lens 105 of the headset. Diffractive, reflective, or refractive optical elements can be placed in the optical system to change the image characteristics. Although this optical path is described in this embodiment, there are many examples of near eye optical relay means used in the technical field of head mounted displays, and those skilled in the art will be able to use this attachment for this attachment. Any number of alternative paths can be designed.

  FIG. 8b shows an attachment that is “fliped up” by hinge 805 so that the user can switch modes without completely removing the attachment. The headset is a means of electrically or optically detecting the presence and position of the attachment so that the firmware and software associated with the system can perform the necessary image correction (such as inversion) to support the mode in use. (Not shown) is assumed. Users can also “flip down” the attachment from a raised position by a quick head movement to switch to enclosure-type virtual reality mode without leaving the keyboard, game controller, or other device. It is also envisaged that mechanical means (not shown) can be provided.

  FIG. 8c shows a front view of the attachment attached by a clamp to the projector in an engaged position covering the front of the headset. This attachment is shown in X-ray style to show the headset behind it, but is considered opaque. One skilled in the art can design many other enclosures and attachment means, such as by magnets, snaps or hook-and-loop fasteners, but in all designs the fixture covers camera 103 or its view. It should not be restricted. Also, none of the present description excludes half of the attachments (shown in FIG. 8d), such as those used for augmented reality applications that supply closed images or information to only one eye.

  It will also be apparent to those skilled in the art of optical relays that equivalent attachments can be designed for the single projector embodiment disclosed in FIG. One such embodiment is accompanied by a beam splitter or active beam switch that relays the image laterally for each eye prior to launching into a system similar to that shown in FIG. 8a. Alternatively, an optical relay may send the output of the projector to both eyes, in which case undesired frames are excluded by a temporal shutter, polarizing filter, spectral filter, or other optical means.

In some augmented reality applications, it is desirable to mix an image generated by a computer graphics system with an actual image of the real world. To achieve this goal, the attachment can embody means for providing a path for light entering from the outside world, as shown in FIG. In this embodiment, the enclosure comprises a lens or lens system 901 attached to the opening and facing forward. A lens or lens system 901 collects external light and passes it through filter means 902 and semi-reflective mirror 804 before entering the coaxial optical path previously described in FIG. 8a. The optical system, such as field of view, anamorphic, color correction, and other characteristics of the projection or external path, can be modified by attachments with refractive, diffractive, and reflective optical elements. Filter means 902 may include a polarizer or electronic shutter, a spectral filter, or other means for masking or blocking a portion of the image collected by the lens or lens system 901. The electronic means for controlling this optical operation are not shown, but are known to those skilled in the art. Alternatively, as shown in FIG. 10, a “see-through” mode may be achieved by attaching one or more cameras 1001 to the front of the enclosure. As is well known in the art, in this embodiment, external world images are electrically relayed to graphical mixed firmware and software (not shown) that controls masking and replacement or overlay of CGI images. (Also not shown). The embodiment of FIG. 10 is particularly useful when combined with image processing software such as those developed to track finger movements and gestures with images returned by a video camera.

CONCLUSION Exemplary embodiments have been described herein by way of example. However, changes and modifications may be made to this embodiment without departing from the true scope and spirit of the elements, products and methods covered by this embodiment as defined by the claims. Will be understood by those skilled in the art.

  US Patent Literature

  US patent application

  Other publications

Claims (20)

In head mounted display,
A headset or eyeglass frame that supports one or more image projectors,
The projector is mounted directly above or below the center line of the vertical pupil, a headset or eyeglass frame;
One or more retroreflective surfaces,
Returning a projected image to the headset; a surface;
And a filter means for reducing the luminance of an undesired image emitted from the projector attached to both sides of the headset. The filter means includes
A first polarizing filter applied to the first projector;
A second polarizing filter applied to a second projector, wherein a direction of polarization of the second filter is orthogonal to a direction of polarization of the first polarizing filter;
A first viewing lens having a polarizing filter,
The first viewing lens is on the same side of the headset as the first projector;
The first view, wherein the first polarizing filter on the first viewing lens is configured to exclude reflected images passing through the second polarizing filter on the second projector. Ing Lens,
A second viewing lens having a polarizing filter, wherein the second viewing lens is on the same side of the headset as the second projector;
The second view, wherein the second polarizing filter on the second viewing lens is configured to exclude reflected images that pass through the first polarizing filter on the first projector. The head mounted display according to claim 1, further comprising an ing lens.   The head mounted display according to claim 2, wherein the type of polarization of the light projected on each side is a planar.   The head mounted display according to claim 2, wherein the type of polarization of light projected on each side is circular. The filter means includes
A first spectral filter applied to the first projector;
A second spectral filter applied to a second projector,
A second spectral filter that passes a portion of the visible spectrum that is not in common with the first spectral filter;
A first viewing lens having a spectral filter comprising:
The first viewing lens is on the same side of the headset as the first projector;
The first view, wherein the first spectral filter in the first viewing lens is configured to exclude reflected images that pass through the second spectral filter in the second projector. Ing Lens,
A second viewing lens having a spectral filter,
The second viewing lens is on the same side of the headset as the second projector;
The second spectral filter in the second viewing lens is configured to exclude reflected images passing through the first spectral filter on the first projector; The head mounted display according to claim 1, further comprising a viewing lens.   6. The head mount according to claim 5, wherein the processing of the spectral filter of the projector is by color selection in encoding pixels of the projected image or by the emission spectrum of the physical illuminant used. display. The filter means includes
The first and second image projectors have alternating time slots for image projection;
First and second viewing lenses having mounted or built-in transparency switching means,
The head mounted display according to claim 1, wherein the switching unit is adjusted to a time slot of the projection so as to block images emitted from projectors on both sides. The filter means includes
An anisotropic retroreflective surface having an anisotropic major axis in a vertical orientation;
Vertically aligning the projector above or below a center position with respect to a plurality of eye positions of the headset;
The head-mounted display according to claim 1, wherein the anisotropic retroreflection has a reflection luminance pattern that is sufficiently narrow in a horizontal dimension so as to separate reflection images. A head-mounted display according to claim 1;
One or more cameras mounted on the headset for receiving light signals from a geometric array of light emitters, the system comprising:
The emitter is mounted with the retroreflective surface, and one of the emitters in the array sends a coded identification pattern.   The system of claim 9, wherein the emitter projects infrared light.   The system of claim 9, further comprising means for calculating a user's line of sight for the array of emitters from the optical signal. Projection augmented reality headset or glasses,
A system comprising a detachable attachment,
The attachment can be attached to one side or both sides of the front part of the headset,
The attachment includes a means for reconfiguring the operation of the headset to the operation of a near eye display system. A lens to receive images from the real world in front of the user;
The attachment of claim 12, further comprising means for mixing the image with an image from a projector.   14. The attachment of claim 13, further comprising means for masking a spatial portion of the image from the real world prior to mixing with the image from the projector. A hinged attachment means,
The attachment according to claim 12, further comprising means for switching the operation mode of the headset by rotating the attachment in and out of a path of image projection by the headset. Further comprising an electrical or optical sensor;
The attachment of claim 15, wherein the sensor includes means for providing information to the firmware or software of the system about the presence, location, or both of the attachment. A method for displaying augmented reality,
Projecting images from one or more head-mounted image projectors;
Reflecting the image back to the headset by one or more retroreflective surfaces;
Filtering the reflected image by means selected from the group consisting of time series, polarization, spectral utilization, or spectral luminance pattern;
Passing the filtered image to a selected user's eye. A method for displaying virtual reality,
Providing a head-mounted projection augmented reality appliance;
Attaching an optical device to the appliance, and
The apparatus wherein the device redirects the projected image to near-eye mode. A method for switching the mode of operation of a head mounted display,
With the process of lowering the optical device without using the hand by whispering the head,
The apparatus wherein the device redirects the projected image to near-eye mode. A method for tracking the line of sight of a head mounted display,
Imaging an asymmetric pattern of five or more infrared light emitting diodes with one or more high resolution electronic cameras;
Fixing the pattern of the emitter according to its position relative to the world position;
Modulating a diode of the emitter having a unique position in the pattern using a unique identification code number;
Encoding the modulation to enable demodulation by image processing of a signal from the camera that performs the imaging;
Extracting the unique identification code number from the demodulation;
Retrieving reference size and shape information stored for the identification number;
Analyzing the image from the camera that performs the imaging with respect to the size and shape stored for the reference pattern to obtain a solution for the coordinates of the line of sight.