JP2020513590A - 3D object rendering using detected features - Google Patents

Abstract

The augmented reality display system is configured to use fiducial markers to align 3D content with real objects. The augmented reality display system can optionally include a depth sensor configured to detect the location of the real object. The augmented reality display system also includes a light source configured to illuminate at least a portion of the object with invisible light, and a light sensor configured to use the reflected portion of the invisible light to form an image. Can be included. The processing circuitry of the display system identifies the location marker based on the difference between the emitted light and the reflected light and determines the orientation of the real object based on the location of the real object and the location of the location marker. can do.

Description

(Incorporated by reference)
The present application discloses U.S. Provisional Application No. 62 / 433,767, filed December 13, 2016, entitled "3D OBJECT RENDING USING DETECTED FEATURES", 35 U.S.S. S. C. Claiming priority under §119 (e), which US provisional application is hereby incorporated by reference in its entirety for all purposes. The present application is further incorporated by reference in its entirety for each of the following patent applications: US Application No. 14 / 555,585 (filing date November 27, 2014); US Application No. 14 / 690,401 (application No. 14 / 212,961 (filing date March 14, 2014); US application No. 14 / 331,218 (filing date July 14, 2014); and US Application No. 15 / 072,290 (filing date March 16, 2016).

  The present disclosure relates to optical devices, including virtual reality and augmented reality imaging and visualization systems.

  Modern computing and display technologies are driving the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or parts thereof are real. Presented to the user in a manner that can be seen or so perceived. Virtual reality, or "VR," scenarios typically involve presentation of digital or virtual image information without transparency to other real-world visual inputs, augmented reality or "AR". Scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. A mixed reality or "MR" scenario is a type of AR scenario that typically involves virtual objects integrated into and responsive to the natural world. For example, in an MR scenario, AR image content may be blocked by an object in the real world, or otherwise perceived to interact with it.

  Referring to FIG. 1, an augmented reality scene 10 is depicted and a user of AR technology sees a real-world park setting 20, featuring people, trees, buildings in the background, and concrete platforms 30. In addition to these items, users of AR technology will also find "virtual" images of robots 40 standing on the real-world platform 30 and flying cartoon-like avatar characters 50, which look like bumblebee anthropomorphisms. Although we perceive "viewing" "content", these elements 40, 50 do not exist in the real world. The human visual perception system is complex and the generation of AR technology that facilitates a comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements is Have difficulty.

  The systems and methods disclosed herein address various challenges associated with AR and VR technologies.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither this summary nor the following detailed description claims to define or limit the scope of the present subject matter.
(Example)
An augmented reality display system configured to match 1. 3D content with real objects, the system comprising:
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct the image to the wearer's eyes;
A light source configured to illuminate at least a portion of the object by emitting invisible light;
A light sensor configured to use the invisible light to image the portion of the object illuminated by the light source;
A processing circuit that relates to the location of an object, the orientation of an object, or both based on one or more characteristics of a feature in an image formed using a reflected portion of invisible light. A processing circuit configured to determine information.
2. The augmented reality display system of Example 1, wherein features in the image formed using the reflected portion of invisible light are invisible to the eye.
3. The augmented reality display system of any of Examples 1-2, wherein the invisible light comprises infrared light.
4. The augmented reality display system of any of Examples 1-3, wherein the invisible light emitted by the light source comprises a beam that forms a spot on the portion of the object.
5. The augmented reality display system of any of Examples 1-3, wherein the invisible light emitted by the light source comprises a light pattern.
6. The augmented reality display system of any of Examples 1-5, wherein the characteristic comprises the location of the feature.
7. The augmented reality display system of any of Examples 1-6, wherein the characteristic comprises the shape of the feature.
8. The augmented reality display system of any of Examples 1-7, wherein the characteristic comprises an orientation of the feature.
9. The augmented reality display system of any of Examples 1-8, further comprising a depth sensor configured to detect the location of real objects in the world.
10. Augmented reality display according to any of the examples 1-9, wherein the processing circuit is configured to determine the difference between the distribution of the emitted invisible light and the distribution of the reflected portion of the invisible light. system.
11. The augmented reality display system according to example 10, wherein the processing circuitry is configured to identify a difference signature based on the determined difference.
12. The augmented reality display system of Example 11, wherein the processing circuitry is configured to determine the orientation of the real object based on the location of the real object and the location of the difference signature.
13. The augmented reality display system of Example 12, wherein the processing circuitry is configured to determine a location of the real object based at least in part on the location of the difference signature.
14. The eyepiece is further disposed on the frame, at least a portion of the eyepiece is transparent, and the transparent portion transmits light from the environment in front of the wearer to the eye of the wearer, Augmented Reality Display according to any of Examples 1-13, which is positioned in front of the wearer's eye when the wearer wears the display system to provide a view of the frontal environment of the wearer. system.
15. An augmented reality display system,
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct the image to the wearer's eyes;
A depth sensor configured to map the surface of a real object within the wearer's field of view,
A light source configured to project light at at least a first wavelength incident on the surface of the real object;
A photodetector configured to form an image using a portion of the light reflected by the surface of the real object;
Processing circuit for determining a light difference marker based at least in part on the reflected portion of the light pattern and rendering the virtual object with a fixed displacement relative to the light difference marker. An augmented reality display system comprising: configured processing circuitry.
16. Rendering a virtual object with a fixed displacement to an optical difference marker
Receiving an initial location for rendering a virtual object to a real object at an initial time;
Determining a fixed displacement based on the distance between the initial location and the optical difference marker;
Detecting the subsequent location of the optical difference marker at a time following the initial time,
Rendering the virtual object with a fixed displacement relative to the detected subsequent location of the optical difference marker, the augmented reality display system according to example 15.
17. Further provided is an eyepiece arranged on the frame, at least a portion of the eyepiece being transparent, the transparent portion transmitting light from the environment in front of the wearer to the wearer's eye, 17. The augmented reality display system of example 15 or 16, which is placed in front of the wearer's eyes when the wearer wears the display system to provide a view of the frontal environment.
18. An augmented reality display system configured to align 3D content with real objects, the system comprising:
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct the image to the wearer's eyes;
A depth sensor configured to map the surface of a real object within the wearer's field of view,
A light source configured to illuminate at least a portion of the object by emitting light;
A light sensor configured to use the emitted light to image the portion of the object illuminated by the light source;
A processing circuit, the processing circuit being configured to determine information about a location of the real object, an orientation of the real object, or both based on one or more characteristics of features in the image of the object. And a processing circuit.
19. 19. The augmented reality display system of example 18, wherein the light emitted by the light source comprises a beam that forms a spot on the portion of the object.
20. The augmented reality display system of example 18, wherein the light emitted by the light source comprises a light pattern.
21. The augmented reality display system of any of Examples 18-20, wherein the characteristic comprises the location of the feature.
22. The augmented reality display system of any of Examples 18-21, wherein the characteristic comprises the shape of the feature.
23. The augmented reality display system of any of Examples 18-22, wherein the characteristic comprises an orientation of the feature.
24. The processing circuit of any of Examples 18-23, wherein the processing circuit is configured to determine a difference between the distribution of emitted light and the distribution of the emitted light reflected from the object. Augmented reality display system.
25. The augmented reality display system of Example 24, wherein the processing circuitry is configured to identify a light difference marker based on the determined difference.
26. The augmented reality display system of Example 25, wherein the processing circuitry is configured to determine the orientation of the real object based on the location of the real object and the location of the light difference marker.
27. 27. The augmented reality display system according to example 25 or 26, wherein the processing circuit is configured to determine the location of the real object based at least in part on the location of the light difference marker.
28. The augmented reality display system of any of Examples 18-27, wherein the depth sensor comprises a laser or an ultrasonic range finder.
29. The augmented reality display system of any of Examples 18-28, wherein the depth sensor comprises a camera.
30. Further provided is an eyepiece arranged on the frame, at least a portion of the eyepiece being transparent, the transparent portion transmitting light from the environment in front of the wearer to the wearer's eye, The augmented reality display system of any of Examples 18-29, which is positioned in front of the wearer's eyes when the wearer wears the display system to provide a view of the frontal environment. ..
31. Augmented reality display system according to any of the previous embodiments, wherein the display is configured to render image content as if the different image content were located at different depths.
32. The augmented reality display system according to example 31, wherein the display includes multi-power optical elements having different powers to provide the different depths.

FIG. 1 illustrates a view of an augmented reality (AR) user through an AR device.

FIG. 2 illustrates an embodiment of a wearable display system.

FIG. 3 illustrates a conventional display system for simulating a three-dimensional image for a user.

FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

5A-5C illustrate the relationship between radius of curvature and focal radius.

FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user.

FIG. 7 illustrates an embodiment of the exit beam output by the waveguide.

FIG. 8 illustrates an example of stacked waveguide assemblies, each depth plane containing an image formed using a plurality of different primary colors.

FIG. 9A illustrates a cross-sectional side view of an embodiment of a set of stacked waveguides, each including an incoupling optical element.

FIG. 9B illustrates a perspective view of the multiple stacked waveguide embodiment of FIG. 9A.

FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B.

FIG. 10 schematically illustrates an augmented reality display system configured to track the orientation of objects in the world using markers.

FIG. 11 illustrates an exemplary method of using markers to track the orientation of an object in the world.

Reference is now made to the figures, where like reference numerals refer to like parts throughout. It should be appreciated that the embodiments disclosed herein generally include optical systems, including display systems. In some embodiments, the display system is wearable, which may advantageously provide a more immersive VR or AR experience. For example, a display containing one or more waveguides (eg, a stack of waveguides) may be configured to be positioned and worn in front of the user's eye or viewer. In some embodiments, a stack of two waveguides (one for each eye of the viewer) may be utilized to provide a different image to each eye.
(Exemplary display system)

  FIG. 2 illustrates an embodiment of wearable display system 60. The display system 60 includes a display 70 and various mechanical and electronic modules and systems for supporting the functions of the display 70. Display 70 may be coupled to frame 80, which is wearable by a display system user or viewer 90 and is configured to position display 70 in front of user 90's eye. The display 70 may be considered eyewear in some embodiments. In some embodiments, the speaker 100 is coupled to the frame 80 and is configured to be positioned adjacent to the ear canal of the user 90 (in some embodiments, another speaker not shown is the other of the users). Located adjacent to the ear canal of the earphone to provide stereo / mouldable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphone is configured to allow a user to provide input or commands to system 60 (eg, voice menu command selection, natural language question, etc.), and / or other. Audio communication with other people (eg, other users of similar display systems). The microphone may also be configured as a peripheral sensor to collect audio data (eg, sound from the user and / or the environment). In some embodiments, the display system may also include a peripheral sensor 120a, which is separate from the frame 80 and is on the body of the user 90 (eg, on the head, torso, limbs, etc. of the user 90). May be attached to. Perimeter sensor 120a, in some embodiments, may be configured to obtain data characterizing a physiological condition of user 90. For example, the sensor 120a may be an electrode.

  With continued reference to FIG. 2, the display 70 is operably coupled to a local data processing module 140 by a communication link 130, such as a wire or wireless connectivity, which is fixedly attached to a frame 80 by a user. Fixedly attached to a helmet or hat worn by the user, incorporated into headphones, or otherwise removably attached to the user 90 (eg, in a backpack configuration, in a belt-coupled configuration), etc. It may be mounted in a configuration. Similarly, the sensor 120a may be operably coupled to the local processor and data module 140 by a communication link 120b, such as a wired conductor or wireless connectivity. The local processing and data module 140 may include a digital memory, such as a hardware processor and non-volatile memory (eg, flash memory or hard disk drive), both of which assist in processing, caching, and storing data. Can be used for. The data may include: a) sensors such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein. For example, data captured from (which may be operably coupled to frame 80, or otherwise attached to user 90), and / or b) optionally for processing or readout, and then to display 70, Contains data that has been obtained and / or processed using remote processing module 150 and / or remote data repository 160 (including data related to virtual content). The local processing and data module 140 may be coupled via a wired or wireless communication link, etc. such that these remote modules 150, 160 are operably coupled to each other and available as resources to the local processing and data module 140. Communication links 170, 180 may be operatively coupled to remote processing module 150 and remote data repository 160. In some embodiments, the local processing and data module 140 includes one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. May be included. In some other embodiments, one or more of these sensors may be attached to frame 80 or stand alone structure that communicates with local processing and data module 140 via a wired or wireless communication path. May be

  With continued reference to FIG. 2, in some embodiments remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 160 may comprise digital data storage facilities, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, remote data repository 160 provides one or more remote servers that provide information to local processing and data module 140 and / or remote processing module 150, eg, information for generating augmented reality content. May be included. In some embodiments, all data is stored and all calculations are performed within the local processing and data modules, allowing for fully autonomous use from remote modules.

  Perception of an image as "3D" or "3-D" can be achieved by providing a slightly different presentation of the image to each eye of a viewer. FIG. 3 illustrates a conventional display system for simulating a three-dimensional image of a user. Two distinct images 190, 200 (one for each eye 210, 220) are output to the user. The images 190, 200 are separated from the eyes 210, 220 by a distance 230 along an optical axis or z-axis parallel to the line of sight of the viewer. The images 190, 200 are flat and the eyes 210, 220 can be focused on the images by taking a single accommodating state. Such a 3-D display system relies on the human visual system to combine images 190, 200 and provide depth and / or scale perception of the combined images.

  However, it should be appreciated that the human visual system is more complex and it is more difficult to provide a realistic perception of depth. For example, many viewers of conventional "3-D" display systems may find such systems uncomfortable or may not perceive a sense of depth at all. Without being limited by theory, it is believed that the viewer of the object may perceive the object as "three-dimensional" due to a combination of convergence / divergence (vergence) and accommodation. Convergence / divergence movements of the two eyes relative to each other (ie, the rotation of the eyes so that the pupils move towards or away from each other, converging the eyes' gaze and fixing the object) , Closely associated with the focusing (or “accommodation”) of the eye's lens and pupil. Under normal conditions, a change in the focus of the lens of the eye or an accommodation of the eye to change the focus from one object to another at a different distance is referred to as "accommodation-convergence / divergent motion reflex" and pupil dilation or Under a relationship known as contraction, it will automatically cause a consistent change in vergence-divergence movement at the same distance. Similarly, changes in vergence and divergence movements will induce matching changes in accommodation of lens shape and pupil size under normal conditions. As described herein, many stereoscopic or "3-D" display systems have slightly different presentations (and therefore slightly different) to each eye such that a three-dimensional perspective is perceived by the human visual system. Image) to display the scene. However, such a system, among other things, merely provides a different presentation of the scene, but when the eye sees all the image information in a single accommodating state, "accommodation-convergence-divergent motion reflex". It works against and is uncomfortable for many viewers. The display system, which provides a better match between accommodation and vergence / divergence movements, creates a more realistic and comfortable simulation of 3D images, increases wear duration and thus diagnostic and therapeutic protocols. Can contribute to compliance.

  FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. Referring to FIG. 4, objects at various distances from the eyes 210, 220 on the z-axis are accommodated by the eyes 210, 220 so that they are in focus. The eyes 210, 220 assume a particular accommodation state to focus the object at different distances along the z-axis. As a result, a particular accommodating state is associated such that an object or part of an object in a particular depth plane is in focus when the eye is accommodating with respect to that depth plane. It can be said to be associated with a particular one of the depth planes 240, which has a different focal length. In some embodiments, a three-dimensional image is simulated by providing different presentations of images for each eye 210, 220 and by providing different presentations of images corresponding to each of the depth planes. You may. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 210, 220 may overlap, for example, as the distance along the z-axis increases. In addition, although shown as flat for ease of illustration, the contours of the depth plane are such that all features in the depth plane are in focus with the eye in a particular accommodating state. It should be appreciated that it can be curved in physical space.

  The distance between the object and the eye 210 or 220 may also change the amount of light divergence from that object as seen by that eye. 5A-5C illustrate the relationship between distance and ray divergence. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in FIGS. 5A-5C, the rays diverge more as the distance to the object decreases. As the distance increases, the rays become more collimated. In other words, it can be said that the light field produced by a point (object or part of an object) has a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. The curvature increases with decreasing distance between the object and the eye 210. As a result, at different depth planes, the divergence of the rays is also different, with the divergence increasing with decreasing distance between the depth plane and the viewer's eye 210. Although only the monocular 210 is illustrated in Figures 5A-5C and other figures herein for clarity of illustration, the discussion regarding the eye 210 may apply to both eyes 210 and 220 of the viewer. Please understand.

  Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes and providing depth perception. As a result, a highly plausible simulation of the perceived depth can be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. Different presentations are separately focused by the viewer's eyes, thereby based on the eye accommodation required to focus on different image features for scenes located on different depth planes, and / or Or it may help to provide depth cues to the user based on the observation of different image features on different depth planes that are out of focus.

  FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. Display system 250 is a stack of waveguides or stacked waveguides that can be utilized to provide three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. Includes tube assembly 260. In some embodiments, the display system 250 is the system 60 of FIG. 2, and FIG. 6 schematically shows some portions of the system 60 in more detail. For example, the waveguide assembly 260 may be part of the display 70 of FIG. It should be appreciated that the display system 250 may be considered a light field display in some embodiments.

  With continued reference to FIG. 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 are configured to transmit image information to the eye with varying levels of wavefront curvature or ray divergence. You may. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image injecting devices 360, 370, 380, 390, 400 may serve as a light source for the waveguides and to inject image information into the waveguides 270, 280, 290, 300, 310. Each may be utilized and configured to disperse incident light across each individual waveguide for output towards the eye 210, as described herein. Light exits the output surfaces 410, 420, 430, 440, 450 of the image input devices 360, 370, 380, 390, 400 and the corresponding input surfaces 460 of the waveguides 270, 280, 290, 300, 310. 470, 480, 490, 500. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be the edge of the corresponding waveguide, or a portion of the corresponding major surface of the waveguide (ie, the world). 510 or one of the waveguide surfaces directly facing the viewer's eye 210). In some embodiments, a single beam of light (eg, a collimated beam) may be injected into each waveguide to output the entire field of the cloned collimated beam, which is , Directed towards the eye 210 at a particular angle (and divergence) corresponding to the depth plane associated with a particular waveguide. In some embodiments, a single one of the image injecting devices 360, 370, 380, 390, 400 may include multiple (eg, three) waveguides 270, 280, 290, 300, 310. It may be associated with and throw light into it.

  In some embodiments, the image injection devices 360, 370, 380, 390, 400 each generate image information for injection into their respective waveguides 270, 280, 290, 300, 310. Yes, it is a discrete display. In some other embodiments, the image input devices 360, 370, 380, 390, 400 may, for example, provide image information via one or more optical conduits (such as fiber optic cables). Output of a single multiplexed display that can be sent to each of 380, 390, 400. It is understood that the image information provided by the image injection devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (eg, different primary colors, as discussed herein). I want to.

  In some embodiments, the light launched into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520, which comprises a light module 530, which comprises a light emitting diode. It may include a light emitter such as (LED). Light from the light module 530 may be directed and modified by a light modulator 540, eg, a spatial light modulator, via a beam splitter 550. The light modulator 540 may be configured to change the perceived intensity of light injected into the waveguides 270, 280, 290, 300, 310. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays.

  In some embodiments, the display system 250 directs light in various patterns (eg, raster scan, spiral scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310. Finally, it may be a scanning fiber display comprising one or more scanning fibers configured to project into the viewer's eye 210. In some embodiments, the illustrated image injection device 360, 370, 380, 390, 400 is adapted to inject light into one or more waveguides 270, 280, 290, 300, 310. The configured single scan fiber or bundle of scan fibers may be represented diagrammatically. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may schematically represent a plurality of scanning fibers or a bundle of scanning fibers, each of which may be a light source. Is configured for injection into the associated one of the waveguides 270, 280, 290, 300, 310. It should be appreciated that the one or more optical fibers may be configured to transmit light from the optical module 530 to the one or more waveguides 270, 280, 290, 300, 310. One or more intervening optical structures are provided between the scanning fiber (s) and the one or more waveguides 270, 280, 290, 300, 310, eg, light exiting the scanning fibers. It should be appreciated that may be redirected into one or more waveguides 270, 280, 290, 300, 310.

  Controller 560 controls the operation of one or more of stacked waveguide assemblies 260, including the operation of image injection devices 360, 370, 380, 390, 400, light source 530, and light modulator 540. . In some embodiments, controller 560 is part of local data processing module 140. The controller 560 adjusts the timing and provision of image information to the waveguides 270, 280, 290, 300, 310, eg, according to any of the various schemes disclosed herein, programming (eg, non-compliant). Instructions in a transitory medium). In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 560 may be part of processing module 140 or 150 (FIG. 2) in some embodiments.

  With continued reference to FIG. 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 270, 280, 290, 300, 310 are each planar, or with another major top surface and major bottom surface and edges extending between those major top and bottom surfaces. It may have a shape (eg, curved). In the illustrated configuration, each of the waveguides 270, 280, 290, 300, 310 redirects light, propagates within each individual waveguide, and outputs image information from the waveguides to the eye 210. May include outcoupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguide. The extracted light may also be referred to as outcoupling light, and the optical element that outcouples the light may also be referred to as light extraction optical element. The extracted beam of light may be output by the waveguide at a location where light propagating in the waveguide strikes the light extraction optical element. Outcoupling optical elements 570, 580, 590, 600, 610 may be, for example, gratings that include diffractive optical features as discussed further herein. For ease of explanation and for clarity of illustration, the waveguide 270, 280, 290, 300, 310 is shown positioned on the bottom major surface of the waveguide, but in some embodiments, the outcoupling optical element 570. , 580, 590, 600, 610 may be located on the top major surface and / or the bottom major surface, and / or the waveguides 270, 280, 290, 300, as discussed further herein. , 310 may be placed directly within the volume. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are formed in layers of material that are attached to a transparent substrate to form the waveguides 270, 280, 290, 300, 310. May be done. In some other embodiments, the waveguide 270, 280, 290, 300, 310 may be a monolithic material component and the outcoupling optical element 570, 580, 590, 600, 610 may be a material component thereof. It may be formed on the surface of and / or inside.

  With continued reference to FIG. 6, as discussed herein, each waveguide 270, 280, 290, 300, 310 outputs light to form an image corresponding to a particular depth plane. Is configured as follows. For example, the waveguide 270 proximate to the eye may be configured to deliver collimated light (cast into such a waveguide 270) to the eye 210. The collimated light may represent the optical infinity focal plane. The next upper waveguide 280 may be configured to deliver collimated light that passes through the first lens 350 (eg, a negative lens) before it can reach the eye 210. Such a first lens 350 interprets the eye / brain as the light emanating from the next upper waveguide 280 originates from a first focal plane closer inward from optical infinity toward the eye 210. To generate some convex wavefront curvature. Similarly, the third upper waveguide 290 passes its output light through both the first lens 350 and the second lens 340 before reaching the eye 210. The combined refracting power of the first lens 350 and the second lens 340 is such that the eye / brain was light coming from the third waveguide 290 next to light coming from the upper waveguide 280. It may be configured to generate another incremental amount of wavefront curvature to be interpreted as originating from a second focal plane that is closer inward from optical infinity towards the person.

  The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack having its output visible to the eye due to the aggregate focal power, which represents the focal plane closest to the person. Deliver through all of the lenses between and. A compensating lens layer 620 is provided on top of the stack to compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light originating from the other world 510 of the stacked waveguide assembly 260. It may be arranged to compensate for the collective power of the lower lens stack 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as waveguide / lens pairs available. Both the outcoupling optical element of the waveguide and the focusing side of the lens may be static (ie, not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

  In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set in the same depth plane, or waveguides 270, 280, 290, 300, 310. May be configured to output images set in the same depth plane, with one set per depth plane. This may provide the advantage of forming tiled images to provide an extended field of view in their depth plane.

  Continuing to refer to FIG. 6, the outcoupling optical elements 570, 580, 590, 600, 610 redirect light from their individual waveguides due to the particular depth planes associated with the waveguides. And may be configured to output this light with an appropriate amount of divergence or collimation. As a result, waveguides with different associated depth planes may have different configurations of outcoupling optical elements 570, 580, 590, 600, 610, which, depending on the associated depth planes. , Outputs light with different divergence. In some embodiments, the light extraction optics 570, 580, 590, 600, 610 may be volumetric or surface features, which may be configured to output light at a specific angle. Good. For example, the light extraction optics 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (eg, structures for forming cladding layers and / or voids).

  In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 form a diffraction pattern or “diffractive optical element” (also referred to herein as “DOE”), It is a diffraction feature. Preferably, the DOE is sufficiently low diffraction such that only a portion of the light of the beam is deflected towards the eye 210 at each DOE intersection, while the rest continue to travel through the waveguide through the TIR. Have efficiency. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, so that for this particular collimated beam bouncing within the waveguide, the eye There is a very uniform pattern of emission and emission towards 210.

  In some embodiments, one or more DOEs may be switchable between an actively diffracting “on” state and a significantly non-diffracting “off” state. For example, the switchable DOE may comprise a layer of polymer dispersed liquid crystal, in which the microdroplets comprise a diffraction pattern in the host medium, the refractive index of the microdroplets being the refractive index of the host material. It may be switched to be substantially matched (in which case the pattern does not significantly diffract incident light), or the microdroplets may be switched to a refractive index that is not matched to that of the host medium (that. In that case, the pattern actively diffracts the incident light).

  In some embodiments, a camera assembly 630 (eg, a digital camera, including visible and infrared light cameras) captures an image of the eye 210 and / or tissue surrounding the eye 210 and detects, for example, user input. And / or may be provided to monitor the physiological condition of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 includes an image capture device and a light source that projects light (eg, infrared light) onto the eye, which light can then be reflected by the eye and detected by the image capture device. May be included. In some embodiments, the camera assembly 630 may be mounted on the frame 80 (FIG. 2) and processes image information from the camera assembly 630, as discussed herein, for example, to a user's. It may be in electrical communication with processing module 140 and / or 150, which may make various decisions regarding physiological conditions. It should be appreciated that information about the user's physiological state may be used to determine the user's behavior or emotional state. Examples of such information include movement of the user and / or facial expressions of the user. The user's behavior or emotional state is then determined using the collected environmental and / or virtual content data to determine a relationship between the behavioral or emotional state, the physiological state, and the environment or virtual content data. It may be triangulated. In some embodiments, one camera assembly 630 may be utilized for each eye and monitor each eye separately.

  Referring now to FIG. 7, an example of an exit beam output by a waveguide is shown. Although one waveguide is shown, other waveguides within the waveguide assembly 260 (FIG. 6) may work as well, with the waveguide assembly 260 including multiple waveguides. I want you to understand. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 strikes the DOE 570, some of the light exits the waveguide as an exit beam 650. The exit beam 650 is shown as generally parallel, but at an angle (eg, divergent exit beam formation), as discussed herein, and depending on the depth plane associated with the waveguide 270. It may be redirected to propagate to the eye 210. The substantially collimated output beam is a guided wave with an outcoupling optical element that outcouples the light to form an image that appears to be set in the depth plane at a far distance from the eye 210 (eg, optical infinity). It should be appreciated that the tube may be shown. Other waveguides or other sets of outcoupling optics may output a more divergent, exiting beam pattern that causes the eye 210 to adjust to a closer distance and focus on the retina. And will be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

  In some embodiments, a full-color image may be formed in each depth plane by overlaying the image on a primary color, eg, each of three or more primary colors. FIG. 8 illustrates an example of stacked waveguide assemblies, each depth plane containing an image formed using a plurality of different primary colors. The illustrated embodiment shows depth planes 240a-240f, but more or less depth is also contemplated. Each depth plane includes three or more primary colors associated with it, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. It may have an image. Different depth planes are indicated in the figure by different numbers for diopters (dpt) following the letters G, R and B. By way of example only, the number following each of these letters indicates the diopter (1 / m), ie the inverse distance of the depth plane from the viewer, and each box in the figure represents an individual primary color image. In some embodiments, the exact location of the depth plane for different primaries may vary to account for differences in the focusing of different wavelengths of light in the eye. For example, different primary color images for a given depth plane may be placed on the depth plane corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort, and / or reduce chromatic aberration.

  In some embodiments, each primary color light may be output by a single dedicated waveguide, so that each depth plane may have multiple waveguides associated with it. In such an embodiment, each box in the figure, including the letters G, R, or B, may be understood to represent an individual waveguide, three waveguides being provided per depth plane. Alternatively, three primary color images are provided per depth plane. Although the waveguides associated with each depth plane are shown adjacent to each other in this figure for ease of illustration, in a physical device all waveguides are one waveguide per level. It should be appreciated that they may be arranged in a stack with tubes. In some other embodiments, multiple primaries may be output by the same waveguide, eg, only a single waveguide may be provided per depth plane.

  With continued reference to FIG. 8, in some embodiments G is green, R is red and B is blue. In some other embodiments, other colors associated with other wavelengths of light may also be used, including magenta and cyan, or one of red, green, or blue. May replace one or more. In some embodiments, the features 320, 330, 340, and 350 may be active or passive optical filters configured to block or select light from the ambient environment into the viewer's eyes. .

  References to a given color of light throughout this disclosure are understood to include light of one or more wavelengths within the range of wavelengths of light perceived by a viewer as that given color. I want you to understand. For example, red light may include one or more wavelengths of light in the range of about 620-780 nm, and green light may include one or more wavelengths of light in the range of about 492-577 nm. Well, blue light may include light at one or more wavelengths that are in the range of about 435-493 nm.

  In some embodiments, the light source 530 (FIG. 6) is configured to emit light at one or more wavelengths outside the visual perception range of the viewer, eg, infrared and / or ultraviolet wavelengths. Good. In addition, the waveguide's internal coupling, external coupling, and other light redirecting structures of display 250 direct this light from the display to the user's eye 210, for example, for imaging and / or user stimulation applications. May be configured to direct and emit.

  Referring now to FIG. 9A, in some embodiments, light impinging on the waveguide may need to be redirected in order to internally couple the light into the waveguide. In-coupling optical elements may be used to redirect and in-couple light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an embodiment of a plurality or sets 660 of stacked waveguides, each including internal coupling optics. Each waveguide may be configured to output light at one or more different wavelengths or at one or more different wavelength ranges. The stack 660 may correspond to the stack 260 (FIG. 6), and the illustrated waveguides of the stack 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310. Good, but the light from one or more of the image injection devices 360, 370, 380, 390, 400 is in the waveguide from a position that requires that the light be redirected for internal coupling. Please understand that it will be put into.

  The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input area on the waveguide), for example, the incoupling optical element 700 includes a major surface (eg, upper side) of the waveguide 670. Inner coupling optical element 710 is disposed on a major surface of waveguide 680 (e.g., an upper major surface) and inner coupling optical element 720 is disposed on a major surface of waveguide 690 (eg, a major surface). , The upper major surface). In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the individual waveguides 670, 680, 690 (specifically, The one or more incoupling optical elements are reflective polarizing optical elements). As shown, the incoupling optical elements 700, 710, 720 may also be disposed on the upper major surface of their respective waveguides 670, 680, 690 (or the top of the next lower waveguide). Well, in particular, those internally coupled optics are transmissive deflection optics. In some embodiments, the incoupling optical elements 700, 710, 720 may be located within the body of the individual waveguides 670, 680, 690. In some embodiments, as discussed herein, the incoupling optical elements 700, 710, 720 selectively re-transmit one or more wavelengths of light while transmitting other wavelengths of light. It is directional and wavelength selective. Although illustrated on one side or at a corner of the individual waveguides 670, 680, 690, the internal coupling optics 700, 710, 720 may, in some embodiments, be the individual waveguides 670, 680, 690. It is to be understood that it may be arranged in other areas of.

  As shown, the incoupling optical elements 700, 710, 720 may be laterally offset from each other. In some embodiments, each incoupling optical element may be offset to receive light without its light passing through another incoupling optical element. For example, each incoupling optical element 700, 710, 720 may be configured to receive light from different image injection devices 360, 370, 380, 390, and 400, as shown in FIG. May be separated from other internal coupling optical elements 700, 710, 720 (eg, laterally spaced) so as to not substantially receive from other internal coupling optical elements 700, 710, 720. ..

  Each waveguide also includes an associated light dispersive element, eg, light dispersive element 730 is disposed on a major surface (eg, upper major surface) of waveguide 670 and light dispersive element 740 is a light guide element. The light dispersive element 750 is disposed on a major surface (eg, upper major surface) of the waveguide 680 and the light dispersive element 750 is disposed on a major surface (eg, upper major surface) of the waveguide 690. In some other embodiments, the light dispersive elements 730, 740, 750 may be disposed on the bottom major surface of the associated waveguide 670, 680, 690, respectively. In some other embodiments, the light dispersive elements 730, 740, 750 may be disposed on both the top major surface and the bottom major surface of the associated waveguide 670, 680, 690, respectively. , Or light dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

  The waveguides 670, 680, 690 may be separated and separated by, for example, a gas, liquid, and / or solid layer of material. For example, as shown, layer 760a may separate waveguides 670 and 680 and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low index material (ie, a material having a lower index of refraction than the material forming the immediate vicinity of waveguides 670, 680, 690). Preferably, the index of refraction of the material forming layers 760a, 760b is greater than or equal to 0.05 or less than 0.10. Compared to the index of refraction of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b provide total internal reflection (TIR) of light through the waveguides 670, 680, 690 (eg, TIR between the top and bottom major surfaces of each waveguide). ) May be used as a cladding layer. In some embodiments, layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include the nearest cladding layer.

  Preferably, the materials forming the waveguides 670, 680, 690 are similar or identical and the materials forming layers 760a, 760b are similar or identical for ease of manufacturing and other considerations. Is. In some embodiments, the material forming the waveguides 670, 680, 690 may differ between one or more waveguides, and / or the material forming layers 760a, 760b is still They may be different while retaining the various refractive index relationships mentioned above.

  With continued reference to FIG. 9A, rays 770, 780, 790 are incident on the set of waveguides 660. It should be appreciated that the light rays 770, 780, 790 may be launched into the waveguides 670, 680, 690 by one or more image launch devices 360, 370, 380, 390, 400 (FIG. 6). .

  In some embodiments, the light rays 770, 780, 790 have different wavelengths or different wavelength ranges that may correspond to different properties, eg, different colors. The rays 770, 780, 790 may also be laterally displaced to different locations corresponding to the lateral locations of the incoupling optical elements 700, 710, 720. Each of the incoupling optical elements 700, 710, 720 deflects incident light such that the light propagates by TIR through a respective one of the waveguides 670, 680, 690.

  For example, the incoupling optical element 700 may be configured to deflect a light ray 770 having a first wavelength or wavelength range. Similarly, the transmitted light ray 780 impinges on and is deflected by the inner coupling optical element 710, which is configured to deflect light of the second wavelength or range of wavelengths. Similarly, light ray 790 is deflected by an incoupling optical element 720 that is configured to selectively deflect light at a third wavelength or range of wavelengths.

  With continued reference to FIG. 9A, the deflected rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the incoupling optical element 700, 710, 720 of each waveguide deflects light into its corresponding waveguide 670, 680, 690 and incouples light into the corresponding waveguide. . Rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the individual waveguides 670, 680, 690 by TIR, and thus are guided therein. For example, the deflection of light rays 770, 780, 790 may be caused by one or more reflective, diffractive, and / or holographic optical elements such as holographic, diffractive, and / or reflective turning features, reflectors, or mirrors. Good. The deflection may, in some cases, be one or more gratings and / or holographic and / or diffractive optical elements that are configured to redirect or redirect light, eg, to be guided in an optical waveguide. It may also be caused by microstructures such as diffractive features within. The rays 770, 780, 790 propagate through the individual waveguides 670, 680, 690 by the TIR guided therein until they strike the corresponding light dispersive elements 730, 740, 750 of the waveguide.

  9B, a perspective view of the multiple stacked waveguide embodiment of FIG. 9A is illustrated. As described above, the incoupling rays 770, 780, 790 are respectively deflected by the incoupling optical elements 700, 710, 720 and then propagated by TIR in the waveguides 670, 680, 690, respectively. , Be induced. The guided light rays 770, 780, 790 then impinge on light dispersive elements 730, 740, 750, respectively. The light dispersive elements 730, 740, 750 may comprise one or more reflective, diffractive, and / or holographic optical elements such as holographic, diffractive, and / or reflective turning features, reflectors, or mirrors. . The deflection may, in some cases, be one or more gratings and / or holographic and / or diffractive optics configured to redirect or redirect light, for example, so that it can be guided using optical waveguides. It may be caused by microstructures such as diffractive features within the element. The rays 770, 780, 790 are deflected, but the rays 770, 780, 790 are still guided in the waveguide in a manner such that the corresponding light dispersive elements 730, 740, Until it hits 750, it propagates through the individual waveguides 670, 680, 690 by the TIR being guided therein. The light dispersive elements 730, 740, 750 deflect the light rays 770, 780, 790 to propagate toward the outcoupling optical elements 800, 810, 820, respectively.

  Outcoupling optical elements 800, 810, 820 are configured to direct light guided in the waveguide, eg, light rays 770, 780, 790, out of the waveguide and toward a viewer's eye. Outcoupling optical elements 800, 810, 820 thus guide the light in order to reduce the effects of total internal reflection (TIR) so that light is not guided in the waveguide and instead exits it. It may be configured to deflect and redirect light guided in the tube, eg, light rays 770, 780, 790 at angles more perpendicular to the surface of the waveguide. Further, these outcoupling optical elements 800, 810, 820 may be configured to deflect and redirect this light, eg, light rays 770, 780, 790, toward a viewer's eye. Thus, the outcoupling optical elements 800, 810, 820 comprise one or more reflective, diffractive, and / or holographic optical elements such as holographic, diffractive, and / or reflective turning features, reflectors, or mirrors. May be. The deflection may, in some cases, be one or more gratings and / or holographic and / or diffractive optics configured to redirect or redirect light, eg, as guided using an optical waveguide. It may be caused by microstructures such as diffractive features within the element. Optical elements 800, 810, 820 may be configured to reflect, deflect, and / or diffract rays 770, 780, 790 to propagate out of the waveguide towards the user's eye.

  In some embodiments, the light dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, OPE deflects or disperses light into outcoupling optical elements 800, 810, 820, and replicates the beam or beams into a larger number of beams that propagate to the outcoupling optical elements. Both forming the beam. As the beam travels along the OPE, a portion of the beam may be split from the beam and travel in a direction orthogonal to the beam, ie in the direction of the outcoupling optical elements 800, 810, 820. Orthogonal splitting of the beam within the OPE can occur repeatedly through the OPE along the path of the beam. For example, the OPE may include a grating with increasing reflectivity along the beam path such that a series of substantially uniform beamlets is produced from a single beam. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EP) or exit pupil expanders (EPE) that direct light to the viewer's eye 210 (FIG. 7). The OPE may be configured to increase the size of the eyebox, eg, along the x direction, and the EPE may be an axis that intersects (eg, is orthogonal to) the axis of the OPE, eg, along the y direction. The eye box may be increased in.

  Thus, referring to FIGS. 9A and 9B, in some embodiments, the set of waveguides 660 includes, for each primary color, a waveguide 670, 680, 690 and an inner coupling optical element 700, 710, 720. It includes light dispersive elements (eg, OPE) 730, 740, 750 and outcoupling optical elements (eg, EPE) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with a void and / or cladding layer between each one. The incoupling optics 700, 710, 720 redirect (or with different incoupling optics that receive different wavelengths of light) incident light into its respective waveguide. The light then propagates into the individual waveguides 670, 680, 690 at an angle that will result in TIR, and the light is guided therein. In the example shown, the light ray 770 (eg, blue light) is polarized by the first incoupling optical element 700 in the manner described above and then continues to propagate in the waveguide being guided therein, Interacts with a light dispersive element (eg, OPE) 730 that is replicated into multiple rays propagating to an outcoupling optical element (eg, EPE) 800. Light rays 780 and 790 (eg, green light and red light, respectively) pass through waveguide 670 and light ray 780 is incident on internal coupling optical element 710 and is deflected thereby. Ray 780 then bounces through waveguide 680 via TIR and is replicated into a plurality of rays that propagate to an outcoupling optical element (eg, EPE) 810, the light dispersive element (eg, OPE). ) Proceed to 740. Finally, the light ray 790 (eg, red light) passes through the waveguide 690 and strikes the internal coupling optical element 720 of the waveguide 690. The light incoupling optical element 720 is by TIR to a light dispersive element (eg, OPE) 750, where the light beam is replicated into a plurality of rays that it propagates by TIR to the outcoupling optical element (eg, EPE) 820. The light beam 790 is deflected to propagate. The outcoupling optics 820 are then finally further replicated to outcouple the light rays 790 to the viewer, who also receives the outcoupled light from the other waveguides 670, 680.

  FIG. 9C illustrates a top-down plan view (or front view) of the multiple stacked waveguide embodiment of FIGS. 9A and 9B. As shown, the waveguides 670, 680, 690 are vertically (with associated light dispersive elements 730, 740, 750 of each waveguide and associated outcoupling optical elements 800, 810, 820). For example, they may be aligned (along the x and y directions). However, as discussed herein, the incoupling optical elements 700, 710, 720 are not vertically aligned. Rather, the incoupling optical elements are preferably non-overlapping (eg, laterally spaced along the x-direction, as seen in the top and bottom views of the front view in this example). Deviations in other directions, such as the y direction, can also be employed. The non-overlapping spatial arrangement facilitates the injection of light into different waveguides from different sources such as different light sources and / or displays on a one-to-one basis, whereby the specific light source can Allows to be uniquely bound to. In some embodiments, arrays that include non-overlapping laterally spatially separated internal coupling optical elements can be referred to as a shift pupil system, and the internal coupling optical elements within these arrays can be Can correspond to the eyes.

  In addition to coupling light out of the waveguide, the outcoupling optical elements 800, 810, 820 also couple the light out as if it originated from an object at a far or closer distance, depth, or depth plane. It may be collimated or divergent. The collimated light matches, for example, light from an object far from the view. Increasingly diverging light matches light from objects that are closer, for example, 5-10 feet or 1-3 feet from the front of the viewer. The natural lens of the eye adjusts when it sees objects closer to the eye, and the brain may sense this accommodation, which in turn also serves as a depth cue. Similarly, by diverging a certain amount of light, the eye will adjust and perceive that the object is at a closer distance. Thus, the outcoupling optical elements 800, 810, 820 can be configured to collimate or diverge the light as if it emitted from a far or near distance, depth, or depth plane. To that end, the outcoupling optical elements 800, 810, 820 may include optical power. For example, the outcoupling optical elements 800, 810, 820, in addition to deflecting or redirecting light out of the waveguide, these holographic, diffractive, and / or reflective optical elements may further collimate or It may include holographic, diffractive, and / or reflective optical elements that may include diverging optical power. Outcoupling optical elements 800, 810, 820 may alternatively or in addition include refractive surfaces that include refractive power to collimate or diverge light. Outcoupling optical elements 800, 810, 820 may thus comprise, for example, diffractive or holographic turning features, as well as refractive surfaces that provide refractive power. Such a refractive surface may also be included in addition to the outcoupling optical elements 800, 810, 820, for example, on top of the outcoupling optical elements 800, 810, 820. In some embodiments, for example, an optical element, such as a diffractive optical element, a holographic optical element, a refractive lens surface, or other structure, is placed against the outcoupling optical element 800, 810, 820 to collimate or It may provide refractive power that causes divergence. A layer with refractive power or a layer with diffractive and / or holographic features, such as a layer with a refractive surface, is arranged, for example, with respect to the outcoupling optical elements 800, 810, 820 and additionally provides a refractive power. obtain. A combination of contributions from both the outcoupling optical elements 800, 810, 820 having refractive power and the additional layers with refractive power or layers with diffractive and / or holographic features, such as layers with refractive surfaces, is also provided. Considered as a possibility.

(Exemplary 3D Content Rendering System and Method)
In various implementations, the augmented reality systems and methods described herein are used to render virtual content, such as virtual objects, that appear to interact with real objects in the world around the wearer. You can In some embodiments, a depth sensor may be used to map the world around the wearer and / or the wearer's body part and the augmented reality system renders 3D virtual content such as objects or graphics. It can be rendered on real objects found in the world. In one example, the virtual content can be a virtual wristwatch rendered on the wearer's wrist. Therefore, while the wearer of the head-mounted augmented reality device is not wearing the real wrist watch, the virtual wrist watch displayed to the wearer through the display of the device may appear to be located on the wearer's wrist. it can. In another example, the virtual content can be a graphic design for display on a real object such as a logo or advertising content to be displayed on the side of the coffee cup.

The location of the real object associated with the virtual content can be tracked by a depth sensor or the like. The virtual content can be displayed at a location determined based on the location of the real object. For example, as the wearer moves the wrist associated with the virtual watch, the device changes the location of the virtual watch as it is displayed to the wearer such that the watch continues to appear on the wearer's wrist. Can be made However, existing depth sensors may not be able to detect the orientation of the real object associated with the virtual content. For example, if the wearer's wrist or coffee cup described above rotates, the information from the depth sensor may not be sufficient to allow the system to distinguish between different symmetrical or near symmetrical orientations of real objects. Can be enough. Thus, while the depth sensor may detect the location of a real object, the system described herein provides two of the features of the real object (sometimes also referred to herein as landmarks or reference features). Sub-tracking may be used to identify more precise location and / or orientation information that the depth sensor cannot detect.
Various systems and methods described herein allow an augmented reality display system to track the location and orientation of an object based on the light reflection and / or scattering properties of visible or invisible markers at or near the surface of the object. May allow you to: In some exemplary embodiments, an augmented reality display system uses a depth sensor to track the position of an object, identify features on the surface of the object, and orient the object based on the location of the feature with respect to the object. Can be determined. A feature (sometimes referred to herein as a fiducial feature or fiducial) is an object so that its orientation can be tracked without applying additional markers separately for the purpose of orientation tracking. Existing features of As will be explained in more detail, the feature or fiducial may be detected using visible or invisible light, such as in the infrared or ultraviolet range. The reference or feature may be a background feature such as a nevus on the wrist or a seam in a coffee cup, or an invisible feature such as one or more veins on the wrist, arm, or hand.

  Reference is now made to FIG. 10, which shows a schematic diagram of various components of an exemplary augmented reality display system 2010 configured to track the location and orientation of real objects, as described herein. . In some embodiments, the augmented reality display system may be a mixed reality display system. As shown, the augmented reality display system 2010 is configured to deliver augmented reality content to a left eye 2001 and a right eye 2002 of a wearer of the augmented reality display system 2010, a left waveguide stack 2005 and a right guide. A frame 64 is included that at least partially surrounds the waveguide stack 2006. The system further includes a depth sensor 28, a light source 26, and a photodetector 24. Tracking system 22 may include a processing module 70, which may control and / or analyze data received from depth sensor 28, photodetector 24, and / or light source 26. Depth sensor 28, photodetector 24, and / or light source 26 can communicate with processing module 70 via data links 76, 78.

  Depth sensor 28 may be configured to detect the shape and location of various objects within the world around the wearer. For example, objects detected by the depth sensor 28 may include walls, furniture, and other items in the room surrounding the wearer, other people or animals near the wearer, trees, shrubs, cars, buildings, and the like. It may include outdoor objects such as objects and / or parts of the wearer's body such as arms, hands, legs, and feet. In various embodiments, the depth sensor is effective in mapping objects in a range of 0.5 meters to 4 meters from the wearer, 1 meter to 3 meters, up to 5 meters, or any other range. Can be The depth sensor can be an optical depth sensor configured to determine depth using infrared light, visible light, or the like. In various embodiments, the depth sensor may include one or more of a laser source, laser range finder, camera, ultrasonic range finder, or other range sensing, imaging, and / or mapping device. Good.

  The photodetector 24 can be configured to detect one or more of infrared light, visible light, ultraviolet light, or other range of electromagnetic radiation. Similarly, the light source 26 may be configured to emit one or more of infrared light, visible light, ultraviolet light, or other range of electromagnetic radiation. In some embodiments, at least a portion of the spectrum emitted by light source 26 will be detectable by photodetector 24. In some designs, the light source 26 may be mounted on a gimbal or other moveable mount so that the direction of emitted radiation can be controlled independently of the orientation of the augmented reality display device 2010. it can. The photodetector 24 can be an imaging device, such as a camera, configured to capture an image of light within at least a portion of the wearer's field of view. In various embodiments, each photodetector 24 can be a camera and may comprise a two dimensional array of photosensors. In some exemplary embodiments, the light source 26 is configured to emit infrared light within a defined wavelength range and the photodetector 24 includes infrared light reflected by an object within the field of view of the photodetector 24. An infrared sensor or infrared light detector is provided that is configured to use light to acquire an infrared image.

  In some cases, features that are not noticeable in visible light produce distinguishable features when illuminated with invisible light such as infrared light or ultraviolet light. For example, veins, which may not be resolvable to the eye, may be clearly resolvable by an infrared camera in response to infrared illumination. Such veins may be used as a reference to identify the object and track its movements, its translations, and / or changes in orientation. Therefore, illuminating an object with light, such as invisible light, can cause otherwise invisible features to be detected by a camera or imaging sensor. Movement of these features, which may be referred to as criteria (or difference signatures, as more fully described below), may include movement of objects such that placement of virtual content with respect to moving objects may be properly placed. Can be tracked. The vein is used as an example, but other features may also be observable using illumination, such as infrared illumination, using an infrared detector. Other features of the skin, for example, may also be tracked using a camera sensitive to suitable wavelengths (eg, IR sensors) to track movement of the object, including changes in rotation or orientation of the object. , IR light (or UV light) may be reflected or absorbed to create a marker, or fiducial. Once the movement and orientation changes of the object are known, the virtual content can be accurately positioned and oriented. Such virtual content that follows the object or is designed to have a fixed location and / or orientation with respect to the object can be rendered appropriately. Similarly, an appropriate view of virtual content can also be provided. Infrared and ultraviolet light are discussed in the examples above, but both visible and invisible light or other wavelengths of electromagnetic radiation can be used.

  In some cases, the illumination may comprise a pattern such as a grid or array. In addition, the image of the pattern projected onto the surface may be compared by the processing module 70 to the emitted pattern of light to determine the difference between the emitted and reflected light. Similarly, the processing module 70 can be configured to identify locally unique differences within the image. The locally unique difference may be caused by the portion of the real object that has a different reflectivity than the surrounding area. For example, a nevus of the arm or hand, lentis, veins, scar tissue, or other structure may have a different reflectance within the imaged wavelength range (eg, infrared or UV) that is different from the reflectance of the surrounding tissue. You can Thus, if an area of scar tissue is present in the illuminated and imaged area, the light difference between the emitted and reflected radiation distributions may include an abnormal area in the shape of the scar tissue area. Such features, referred to herein as difference signatures, can be used to track an object so that the virtual content can be properly located and oriented with respect to the object.

  With reference to both FIGS. 10 and 11, an exemplary method 1100 of tracking the location and / or orientation of a 3D object using detected features will be described herein. Method 1100 may be implemented by any of the systems described herein, such as the wearable augmented reality display systems 60, 2010 depicted in FIGS. 2 and 10. The method 1100 begins at block 1110, where virtual content is received. In one example, the virtual content can be a 3D rendering of a watch, which can act as a virtual watch visible to the wearer of the augmented reality display system 2010. The system 2010 also receives the location of the virtual content for the detected real object. For example, the system 2010 may use the depth sensor 28 to detect the wearer's wrist, either automatically or based on an indication by the wearer. The location for the virtual watch may be determined automatically or may be indicated by the wearer, such as by a gesture or using any suitable input device. For example, the wearer may select the position of the virtual watch so that the virtual watch appears to be placed around the wearer's wrist, much like a real-world wrist watch. In another example, the virtual content is intended to behave as if attached to a virtual name tag or item identifier, an advertising graphic to be displayed on a real object such as a coffee cup, or a real object. It may be any other type of virtual content. Once the desired location of the virtual content is determined with respect to the real object, the relative location may be selected, aligned, or otherwise finalized. After the virtual content and its location for the real object are received, the method 1100 continues at block 1120.

  At block 1120, the system 2010 emits a radiation pattern and determines a difference between images of the pattern projected on the object. This difference depends on the structural features of the real object and may indicate it. Differences may indicate variations in absorptance, reflectance, and / or scattering due to, for example, structural variations in the object. This difference can be used as a reference, difference signature, or marker to track movement and / or orientation changes. The radiation pattern is emitted by the light source 26 and may be a light pattern such as a textured light field, a grid, or a series of dots, crosses, circles (eg concentric circles or contours), or other patterns. Although patterns such as grids are discussed above, the illumination may also comprise substantially uniform illumination or spots of any shape (eg, circular, square, etc.) and still the difference signature or marker Can be acquired. The emitted radiation pattern can comprise visible or invisible light, such as infrared light, ultraviolet light, or any other suitable wavelength or range of wavelengths. In some embodiments, invisible wavelength ranges such as infrared or ultraviolet may be desirable to avoid distracting the wearer or others by projecting a visible light pattern or light. The direction of the radiation pattern may be selected such that at least part of the radiation pattern is incident on the surface of the real object. For example, in some configurations, the light source is moveable, such as via a gimbal mount or other rotating stage and / or a stage that may be tilted. In some embodiments, the radiation may be projected onto a location closest to the virtual object's received location. In an exemplary implementation of a virtual watch, the radiation may be projected onto the back of the wearer's hand or the wearer's forearm adjacent to the location of the virtual watch. In other embodiments, the radiation may be projected onto a location spaced from the location of the virtual object.

  After the radiation pattern is emitted, a portion of the emitted radiation may be reflected by the real object into the augmented reality display system 2010. A portion of the emitted radiation may be reflected by an outer surface or internal structure below the surface of the object. For example, if the radiation pattern is an infrared light pattern directed at the user's arm or hand, some of the infrared light may be transmitted by the outer skin surface, the inner area of the skin, and / or veins or other structures under the skin. Can be reflected. The reflected portion of the radiation pattern may be detected at photodetector 24. For example, the light source 26 may be an infrared source that emits an infrared radiation pattern, and the photodetector 24 may be an infrared camera configured to acquire an image within the infrared spectrum. Therefore, when a portion of the emitted radiation pattern is reflected by the display system 2010, the photodetector 24 can obtain an image of the reflected light pattern.

  The image of the reflected light pattern can be compared to the distribution of the emitted radiation pattern to determine the light difference signature. The determination of the optical difference signature can occur in the processing module 70 or any other local or remote processing circuitry associated with the augmented reality display system 2010. The processing module 70 can look for unique differences between the emitted and reflected light patterns that can be used as landmarks, markers, or fiducials. If a unique difference is found between the emitted and reflected light patterns, the difference is stored and the light difference signature or marker can also be recorded. In some cases, the unique difference may be caused by a portion of the real object that has a different reflectance than the surrounding area, such as a nevus of the arm or hand, a mole, a vein, scar tissue, or other structure. For example, if an area of scar tissue is present within the area to be illuminated and imaged, the optical difference between the emitted and reflected radiation distributions may serve as a landmark or difference signature. The radiation pattern is unique because various types of biological tissue, such as skin, fat, oxygenated blood, deoxygenated blood, etc., may have different infrared absorption and scattering properties. If the difference cannot be detected at the first wavelength, it may include multiple wavelengths of light). If a detectable and / or locally unique sub-region of a light difference is found between the emitted and reflected light patterns, this difference can be stored as a light difference signature. If no unique difference is found after comparing the emitted and reflected light patterns, which can be used as a landmark or reference, the block 1120 directs the emission pattern to different locations on the real object, eg, different virtual object locations. It can be repeated by emitting to a location adjacent to the portion and / or to a location that is slightly further away from the virtual object location. If the unique difference, which may be used as a landmark, is identified and stored as an optical difference signature or marker, method 1100 continues at block 1130.

  At block 1130, the location of the virtual content is determined for the optical difference signature, landmark, or “difference marker”. The location of the light difference signature or landmark or difference marker can serve as a reference point for rendering the virtual content. The displacement can then be determined with respect to the reference point with respect to the virtual object. For example, the displacement can include coordinates in one or more dimensions. In some embodiments, the displacement can be a location in a 2D or 3D coordinate system with the origin as a reference point. Thus, if the same optical difference signature or marker or landmark is detected again, the virtual content can be rendered in the same place for the optical difference signature, marker or landmark.

  In some implementations, the optical difference signature, marker, or landmark may also be associated with a physical reference point in the real world that can be detected by the depth sensor 28. The physical reference point may be a feature of a real object, such as a finger or wrist bone, in the virtual wristwatch embodiment described herein. The location of the optical difference signature, marker, or landmark, relative to the physical reference point, as well as the location of the virtual object for the optical difference signature, marker, or landmark, can include coordinates in one or more dimensions. The displacements between the virtual object and the light difference markers or landmarks and between the physical reference points and the light difference markers or landmarks can be recorded in the same or different reference frames or coordinate systems.

After the virtual content and physical reference point locations have been determined for the light difference markers or landmarks, the system 2010 uses the depth sensor 28 to intermittently or continuously monitor the physical reference point locations. can do. In some embodiments, the depth sensor 28 may require less power and / or processing capacity than the photodetector 24 and the light source 26 for continuously monitoring the location and / or physical reference points of real objects. .. Thus, the depth sensor 28 may continuously monitor the location of the physical reference point, while the emission and detection of the radiation or radiation pattern by the light source 26 and the photodetector 24 detects changes at the location of the physical reference point. Can be used infrequently, such as when a frame of virtual content needs to be refreshed, or at any other regular or irregular interval. The method 1100 continues at block 1140 when the virtual content rendering location is to be refreshed.
At block 1140, the radiation or radiation pattern is emitted again. The radiation emitted at block 1140 can be co-distributed (eg, uniform distribution, spots, patterns, textures, or equivalent) with the radiation emitted at block 1120. The direction of radiation of the radiation pattern is such that the emitted radiation pattern is at least partially incident on the area of the real object where the optical difference signature, marker, or landmark is expected to be found. It can be determined based on the location of the physical reference point as tracked by 28. The radiation direction can be selected by pivoting or otherwise moving the light source 24 on a gimbal or other movable mounting platform. For example, in an embodiment of the wearer's arm, the emitted radiation pattern is based on a calculated displacement from a physical reference point, which is a nevus, vein, or nevus associated with an optical difference signature, marker, or landmark. It can be directed to other feature locations. The radiation or radiation pattern can be emitted by the light source 24 in the same manner as described above with reference to block 1120. After the radiation or radiation pattern is emitted, the method 1100 continues at block 1150.

  At block 1150, the system 2010 obtains the reflected radiation distribution or pattern and locates the difference signature, marker, or landmark within the reflected radiation distribution or pattern. The reflected radiation or pattern can be detected and / or imaged by a photodetector 26 that is configured to detect light of one or more wavelengths emitted by the light source 24. Differences can then be determined and / or calculated between the emitted and reflected radiation patterns. The stored optical difference signature or marker or landmark is compared to the difference between the emitted and reflected radiation distribution or pattern to verify the presence of the optical difference signature, marker or landmark in the newly reflected radiation distribution, You can determine its location. The location of the light difference signature, marker, or landmark can then be recorded when the virtual content is refreshed to serve as a reference point for the rendering of the virtual content. After the light difference marker is located within the reflected radiation distribution or difference, the method 1100 continues at block 1160.

At block 1160, the virtual content is rendered to the location of the light difference signature, marker, or landmark. The location for rendering the virtual content may be determined based on the displacement between the optical difference signature, the marker, or the landmark and the virtual content calculated at block 1130. By using the same displacement calculated previously, the augmented reality display system 2010 can achieve the advantage of displaying virtual content that appears to move with the associated real object. For example, by repeatedly rendering a virtual wristwatch at the same displacement relative to a light difference signature, marker, or landmark caused by a nevus or vein group on the wearer's hand, the wearer rotates his wrist, or another Thus, even if the orientation of the wrist is modified, the face of the virtual wristwatch appears to remain adjacent to the upper side of the wearer's wrist in a manner that cannot be reliably detected by the depth sensor 28 alone. be able to. After the virtual content is rendered to the optical difference signature, marker, or landmark, the method 1100 can return to block 1140, where blocks 1140-1160 indicate when the virtual content frame should be refreshed. It may always be repeated indefinitely at regular or irregular intervals, if desired. For example, the method 1100 can be performed at any time when a change in physical reference point location is detected, or every second, 5 times / second, 10 times / second, 50 times / second, 100 times / second, or any other. At regular intervals, such as the preferred intervals of, the block 1140 may be returned to. In some embodiments, the light source 24 is configured to continuously project an emitted light, such as infrared light, rather than transmitting discrete pulses of light each time the difference marker is to be relocated. May be.
As discussed above, the systems, devices, methods, and processes discussed with respect to illuminating patterns of light also apply to projection of spots of any shape, such as uniform illumination or circular spots. In addition, although the above discussion referred to the identification and use of markers or landmarks, multiple markers or landmarks may also be identified and used to provide tracking and placement of virtual image content.

  Innovative aspects may be implemented in or associated with various applications and are therefore considered to include a wide variety of variations. For example, variations in EPE shape, number, and / or refractive power are considered. The structures, devices, and methods described herein may find use in displays, such as wearable displays (eg, head-mounted displays), that may be used for augmentation and / or virtual reality, among others. More generally, the described embodiments, whether under motion (such as video) or stationary (such as still images), are textual, graphic, or photographic. , Or may be implemented in any device, apparatus, or system that may be configured to display images. However, the described embodiments include, but are not limited to, mobile phones, multimedia internet enabled cellular phones, mobile television receivers, wireless devices, smartphones, Bluetooth® devices, personal digital assistants (PDAs), wireless electronic devices. Email receivers, handheld or portable computers, netbooks, notebooks, smartbooks, tablets, printers, copiers, scanners, fax devices, Global Positioning System (GPS) receivers / navigators, cameras, digital media players (MP3 players) Etc.), camcorders, game consoles, watches, table clocks, calculators, TV monitors, flat panel displays, electronic reading devices (eg electronic readers), computer monitors, automatic displays. B (including odometer and speedometer display, etc.), cockpit control and / or display, camera view display (display of rear camera in vehicle, etc.), electronic photograph, electronic billboard or sign, projector, architectural structure, microwave oven, Refrigerator, stereo system, cassette recorder or player, DVD player, CD player, VCR, wireless, portable memory chip, washing machine, dryer, washer / dryer, parking meter, head-mounted display, and various imaging systems It is contemplated that it may be included in or associated with various electronic devices of the. Thus, the present teachings are not limited to only the embodiments depicted in the figures, but are instead intended to have broad availability, as will be readily apparent to those skilled in the art.

  Various modifications of the embodiments described in this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be practiced in other implementations without departing from the spirit or scope of the disclosure. It may be applied to the form. Therefore, the claims are not intended to be limited to the embodiments presented herein and are considered to be the broadest consistent with the disclosure, principles, and novel features disclosed herein. Should be. The word "exemplary" is used exclusively herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. In addition, those skilled in the art will recognize that the terms "upper" and "lower", "upper" and "lower", etc. are sometimes used for ease of illustration and on properly oriented pages. It will be readily understood that relative positions corresponding to the orientations of the figures are shown and may not reflect the proper orientation of the structures described herein as they are implemented.

  Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Further, features are described above as operating in some combinations, and further, although initially claimed as such, one or more features from the claimed combination may, in some cases, come from the combination. The combinations that can be deviated and claimed may be sub-combinations or variations of sub-combinations.

  Similarly, although acts are depicted in the drawings in a particular order, this may be done in the particular order in which such acts are shown or in a sequential order to achieve the desired result, or , Should not be understood to require that all illustrated acts be performed. Moreover, the drawings may diagrammatically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated within the exemplary process illustrated graphically. For example, one or more additional acts may be performed before, after, concurrently with, or during any of the depicted acts. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems generally It should be appreciated that they can be integrated together in a single software product or packaged in multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

  Various exemplary embodiments of the invention are described herein. In a non-limiting sense, reference is made to these examples. They are provided to illustrate the broader applicable aspects of the invention. Various changes may be made to the invention described and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act or step, to the objective, spirit or scope of the present invention. Moreover, as will be appreciated by one of skill in the art, each individual variation described and illustrated herein has discrete components and features, which do not depart from the scope or spirit of the invention. It may be easily separated from, or combined with, the features of any of some other embodiments. All such modifications are intended to be within the scope of the claims associated with this disclosure.

The present invention includes methods that may be performed using the device. The method may include the act of providing such a suitable device. Such provision may be done by the end user. In other words, the act of "providing" simply means that the end user obtains, accesses, approaches, positions, configures, activates, turns on, and so on, to provide the required devices in the method. Or demand to act differently. The methods described herein may be performed in any order of the recited events that is logically possible and in the recited order of events.

  Exemplary aspects of the invention are described above, along with details regarding material selection and manufacturing. With respect to other details of the invention, these are understood in relation to the above referenced patents and publications, and may generally be grasped or understood by those skilled in the art. The same may be true with respect to the method-based aspects of the invention in terms of additional acts as commonly or theoretically employed.

  In addition, while the present invention has been described with reference to several embodiments, which optionally incorporate various features, the present invention is described or indicated as considered with respect to each variation of the invention. It is not limited to one. Various modifications may be made to the invention as described and without departing from the true spirit and scope of the invention (either described herein or included for the sake of somewhat simplicity). Equivalents (whether or not) may be substituted. In addition, where a range of values is provided, all intervening values between the upper and lower limits of the range, and any other stated or intervening value within the specified range, are encompassed within the invention. Be understood.

  Also, any optional feature of the variations of the invention described herein, either independently, or in combination with any one or more of the features described herein, is described. And that it may be billed. Reference to a singular item includes the possibility that there are plural of the same items present. More specifically, as used herein and in the claims associated therewith, the singular forms "a, an", "said, the" are particularly different. Unless otherwise noted, includes multiple referents. In other words, the use of articles allows for "at least one" of the subject item in the above description and claims associated with this disclosure. Further, it is noted that such claims may be drafted to exclude any optional element. Accordingly, this description is used as an antecedent for the use of such exclusive terms, such as "only", "only", and the like, or the use of "negative" restrictions, in connection with the description of the claim elements. The purpose is to fulfill the function of.

  The term “comprising”, in a claim associated with this disclosure, without the use of such exclusive terms, refers to a given number of elements being recited in such claim or the addition of features. Shall allow the inclusion of any additional element (s) regardless of whether they may be considered to transform the nature of the elements recited in such claims. Unless defined otherwise herein, all technical and scientific terms used herein have the broadest possible general understanding while maintaining the validity of the claims. Is given the meaning of being.

  The scope of the present invention should not be limited to the examples provided and / or the specification, but rather should be limited only by the scope of the claims associated with this disclosure.

Claims (32)

An augmented reality display system configured to align 3D content with real objects, the system comprising:
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct an image to the wearer's eyes;
A light source configured to illuminate at least a portion of the object by emitting invisible light;
A photosensor configured to image the portion of the object illuminated by the light source using the invisible light;
A processing circuit, wherein the processing circuit is based on one or more characteristics of features in the image formed using the reflected portion of the invisible light, the location of the object, the orientation of the object. , And processing circuitry configured to determine information about both or both.   The augmented reality display system of claim 1, wherein features in the image formed using the reflected portion of the invisible light are invisible to the eye.   The augmented reality display system of claim 1-2, wherein the invisible light comprises infrared light.   The augmented reality display system of any of claims 1-3, wherein the invisible light emitted by the light source comprises a beam that forms a spot on the portion of the object.   The augmented reality display system of any of claims 1-3, wherein the invisible light emitted by the light source comprises a light pattern.   The augmented reality display system of any of claims 1-5, wherein the characteristic comprises a location of the feature.   The augmented reality display system of any of claims 1-6, wherein the characteristic comprises a shape of the feature.   The augmented reality display system of any of claims 1-7, wherein the characteristic comprises an orientation of the feature.   9. The augmented reality display system of any of claims 1-8, further comprising a depth sensor configured to detect the location of real objects in the world.   10. The extension according to any of claims 1-9, wherein the processing circuit is configured to determine a difference between the distribution of the emitted invisible light and the distribution of the reflected portion of the invisible light. Reality display system.   The augmented reality display system of claim 10, wherein the processing circuitry is configured to identify a difference signature based on the determined difference.   The augmented reality display system of claim 11, wherein the processing circuitry is configured to determine an orientation of the real object based on the location of the real object and the location of the difference signature.   The augmented reality display system of claim 11, wherein the processing circuitry is configured, at least in part, to determine the location of the real object based on the location of the difference signature.   Further comprising an eyepiece arranged on the frame, at least a portion of the eyepiece is transparent, the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes, 14. Any of claims 1-13, wherein the wearer is positioned in front of the wearer's eyes when wearing the display system to provide a view of the wearer's frontal environment. The described augmented reality display system. An augmented reality display system,
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct an image to the wearer's eyes;
A depth sensor configured to map the surface of a real object within the wearer's field of view;
A light source configured to project light at at least a first wavelength incident on the surface of the real object;
A photodetector configured to form an image using a portion of the light reflected by the surface of the real object;
Processing circuit for determining a light difference marker based at least in part on the reflected portion of the light pattern and rendering a virtual object with a fixed displacement relative to the light difference marker. An augmented reality display system comprising: Rendering the virtual object with a fixed displacement with respect to the optical difference marker,
Receiving an initial location for rendering the virtual object to the real object at an initial time;
Determining the fixed displacement based on a distance between the initial location and the optical difference marker;
Detecting a subsequent location of the light difference marker at a time following the initial time;
16. Rendering the virtual object at the fixed displacement with respect to the detected subsequent location of the light difference marker.   Further comprising an eyepiece arranged on the frame, at least a portion of the eyepiece is transparent, the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes, 17. The extension of claim 15 or 16 positioned at a location in front of the wearer's eyes when the wearer wears the display system to provide a view of the wearer's front environment. Reality display system. An augmented reality display system configured to align 3D content with real objects, the system comprising:
A frame configured to be mounted on the wearer,
An augmented reality display attached to the frame and configured to direct an image to the wearer's eyes;
A depth sensor configured to map the surface of a real object within the wearer's field of view;
A light source configured to illuminate at least a portion of the object by emitting light;
A light sensor configured to use the emitted light to image the portion of the object illuminated by the light source;
Processing circuitry for determining information about the location of the real object, the orientation of the real object, or both based on one or more characteristics of features in the image of the object. And a processing circuit configured.   19. The augmented reality display system of claim 18, wherein the light emitted by the light source comprises a beam that forms a spot on the portion of the object.   19. The augmented reality display system of claim 18, wherein the light emitted by the light source comprises a light pattern.   21. The augmented reality display system of any of claims 18-20, wherein the characteristic comprises the location of the feature.   The augmented reality display system of any of claims 18-21, wherein the characteristic comprises a shape of the feature.   The augmented reality display system of any of claims 18-22, wherein the characteristic comprises an orientation of the feature.   24. Any of claims 18-23, wherein the processing circuit is configured to determine a difference between the distribution of the emitted light and the distribution of the emitted light reflected from the object. Augmented reality display system.   The augmented reality display system of claim 24, wherein the processing circuitry is configured to identify a light difference marker based on the determined difference.   26. The augmented reality display system of claim 25, wherein the processing circuitry is configured, at least in part, to determine an orientation of the real object based on the location of the light difference marker.   27. The augmented reality display system of claim 25 or 26, wherein the processing circuitry is configured to determine the location of the real object based at least in part on the location of the light difference marker.   The augmented reality display system of any of claims 18-27, wherein the depth sensor comprises a laser or an ultrasonic range finder.   The augmented reality display system of any of claims 18-28, wherein the depth sensor comprises a camera.   Further comprising an eyepiece arranged on the frame, at least a portion of the eyepiece is transparent, the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes, 30. Any of claims 18-29, positioned at a location in front of the wearer's eyes when the wearer wears the display system so as to provide a view of the wearer's frontal environment. The described augmented reality display system.   The augmented reality display system of any of the preceding claims, wherein the display is configured to render image content as if the different image content were located at different depths.   29. The augmented reality display system of claim 28, wherein the display includes multi-power optical elements having different powers to provide the different depths.