CN114761859A

CN114761859A - Augmented eye tracking for augmented or virtual reality display systems

Info

Publication number: CN114761859A
Application number: CN202080081999.0A
Authority: CN
Inventors: N·阿伯莱; E·契瓦拉兹; P·德戈尔; J·加梅特; G·科森迪; S·A·甘珀
Original assignee: Magic Leap Inc
Current assignee: Magic Leap Inc
Priority date: 2019-11-26
Filing date: 2020-11-23
Publication date: 2022-07-15
Also published as: EP4066044A1; JP2024045580A; US11681362B2; JP7467623B2; US20210157401A1; EP4066044A4; JP2023502337A; WO2021108327A1

Abstract

Methods and display systems for augmented eye tracking for display systems, such as augmented or virtual reality display systems, are described herein. The display system may include: a light source configured to output light; and a movable diffraction grating configured to reflect light from the light source, the reflected light forming a light pattern on the eye of the user; a plurality of photodetectors that detect light reflected from the eye; and one or more processors. The display system changes the orientation of the diffraction grating such that the light pattern reflected from the diffraction grating is scanned along an axis on the eye. The light intensity pattern is obtained via a light detector, wherein the light intensity pattern represents a light detector signal obtained by detecting light reflected from the eye while scanning the light pattern over the eye. Due to differences in how light is reflected by different parts of the eye, different eye poses provide different light intensity patterns, and the eye pose is determined based on the detected light intensity patterns.

Description

Augmented eye tracking for augmented or virtual reality display systems

Priority requirement

This application claims priority from us provisional patent application 62/940,785 entitled "ENHANCED EYE TRACKING FOR AUGMENTED OR virtuall REALITY DISPLAY SYSTEMS (FOR AUGMENTED eye tracking FOR AUGMENTED OR VIRTUAL reality display systems)" filed on 26/11/2019.

Is incorporated by reference

The present application is incorporated by reference in its entirety in U.S. application No.15/469,369 filed 24.3.2017 and published 28.9.2017 as U.S. patent application publication No. 2017/0276948.

Technical Field

The present disclosure relates to display systems, and more particularly to augmented and virtual reality display systems.

Background

Modern computing and display technology has helped in the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner in which they appear to be, or can be perceived as, real. Virtual reality or "VR" scenes typically involve the presentation of digital or virtual image information, while being opaque to other real-world visual inputs; augmented reality or "AR" scenes typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or "MR" scenes are a type of AR scene that generally involves virtual objects that are integrated into and respond to the natural world. For example, an MR scene may include AR image content that appears to be blocked by or otherwise perceived as interacting with an object in the real world.

Referring to fig. 1, an augmented reality scene 10 is depicted. A user of AR technology sees a real world park-like setting 20 featuring people in the background, trees, buildings and a concrete platform 30. The user also perceives that he/she "sees" virtual content, such as the robotic statue 40 standing on the real world platform 30 and the flying avatar 50 in the form of a cartoon, which appears to be an avatar for bumblebee. These

elements

50, 40 are "virtual" in that they do not exist in the real world. Since the human visual perception system is complex, it is challenging to produce a comfortable, natural-feeling, rich-appearing AR technique that facilitates virtual image elements among other virtual or real-world imagery elements.

The systems and methods disclosed herein address various challenges related to display technology, including AR and VR technology.

Disclosure of Invention

In some embodiments, a display system configured to present virtual content to a user is provided. The display system includes: a light source configured to output light; a movable reflector configured to reflect the output light to an eye of the user to scan a pattern formed by the light on the eye; a plurality of light detectors configured to detect reflections of light scanned over the eye; and one or more processors configured to perform operations. The operations include causing an orientation of the movable reflector to be adjusted such that the reflected light is scanned over the eye. Obtaining, via the photodetectors, respective light intensity patterns, wherein the light intensity patterns represent photodetector signals at different times, and the photodetector signals are obtained during scanning of the reflected light on the eye. Determining an eye pose of the eye based on the light intensity pattern, the eye pose representing an orientation of the eye.

In some embodiments, a method implemented by a display system of one or more processors is provided. The display system is configured to present virtual content to a user based at least in part on eye gestures of the user's eyes. The method includes adjusting a position of a light pattern directed onto the eye such that the light pattern moves over the eye. A plurality of light intensity patterns are obtained, the light intensity patterns representing photodetector signals at different times, the photodetector signals obtained from respective photodetectors during adjustment of the position of the light patterns. Determining the eye pose of the eye based on the light intensity pattern, the eye pose representing an orientation of the eye.

In some embodiments, a non-transitory computer storage medium is provided. The non-transitory computer storage medium stores instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations. The operations include adjusting a position of a light pattern directed onto the eye such that the light pattern moves over the eye. A plurality of light intensity patterns are obtained, the light intensity patterns representing photodetector signals at different times, the photodetector signals obtained from respective photodetectors during adjustment of the position of the light patterns. Determining the eye pose of the eye based on the light intensity pattern, the eye pose representing an orientation of the eye.

In some embodiments, a display system configured to present virtual content to a user is provided. The display system includes: a light source configured to output light; a movable reflector configured to reflect the output light to an eye of the user to scan a pattern formed by the light on the eye; a plurality of light detectors configured to detect reflections of light scanned over the eye; and one or more processors configured to perform operations. The operations include obtaining respective light intensity patterns via the photodetectors, wherein the light intensity patterns represent photodetector signals at different times, the photodetector signals obtained during scanning of the reflected light on the eye. Based on the light intensity pattern, one or both of a size and a position of a physiological feature of the eye is determined.

Additional examples of embodiments are provided below.

An example 1. a display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

a movable reflector configured to reflect the output light to an eye of the user to scan a pattern formed by the light on the eye;

A plurality of light detectors configured to detect reflections of light scanned over the eye; and

one or more processors configured to perform operations comprising:

causing adjustment of an orientation of the movable reflector to cause scanning of the reflected light over the eye;

obtaining, via the photodetectors, respective light intensity patterns, wherein the light intensity patterns represent photodetector signals at different times, the photodetector signals obtained during scanning of the reflected light on the eye; and

determining an eye pose of the eye based on the light intensity pattern, the eye pose representing an orientation of the eye.

Example 2. the display system of example 1, wherein the light source is a diode.

Example 3. the display system of example 2, wherein the diode is a vertical cavity surface emitting laser.

Example 4. the display system of example 1, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light rays that span an area of the eye.

Example 5 the display system of example 1, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light beams.

Example 6 the display system of example 1, wherein the movable reflector comprises a plurality of diffraction gratings, each diffraction grating configured to form a different light pattern for scanning over the eye.

Example 7. the display system of example 1, wherein the movable reflector is a micro-electromechanical system (MEMS) mirror.

Example 8 the display system of example 1, wherein the light detector is a photodiode, and wherein each light intensity pattern represents a plot of current versus position information associated with the position of the movable reflector.

Example 9. the display system of example 8, wherein the diffraction grating is located on or forms part of a MEMS mirror, and wherein the position information indicates an orientation of the MEMS mirror, the MEMS being adjustable by the display system.

Example 10. the display system of example 1, wherein the light source is one of two light sources configured to output light to the movable reflector, wherein each of the light sources is configured to form a respective portion of the light pattern.

Example 11 the display system of example 1, wherein the light detector is a photodiode, and wherein each light intensity pattern represents a plot of current versus time.

Example 12. the display system of example 1, wherein the light pattern defines a V-shape extending from a lower portion of the eye to an upper portion of the eye.

Example 13 the display system of example 1, wherein the light forming the light pattern comprises multi-colored light.

Example 14. the display system of example 13, wherein the light pattern includes two portions extending in different directions.

Example 15. the display system of example 14, wherein each of the two portions is formed from a different color of light.

Example 16. the display system of example 14, wherein the two portions are configured to extend across a vertical axis of the eye, wherein the two portions extend in opposite directions along a horizontal axis to form a V-shape.

Example 17. the display system of example 1, wherein the light pattern comprises a plurality of consecutive rows of light.

Example 18 the display system of example 17, wherein different rows of light include light beams having different amounts of divergence.

Example 19 the display system of example 18, wherein a row of light comprises a converging light beam, wherein another of the rows of light comprises a collimated light beam.

Example 20 the display system of example 18, wherein the row of light comprises a diverging beam of light.

Example 21. the display system of example 17, wherein the row of light defines an angle of less than 90 ° relative to a horizontal axis of the eye.

Example 22. the display system of example 1, wherein the position of the light detector defines a corner of a rectangle about the eye.

Example 23. the display system of example 1, wherein the light detector defines a linear array of light detectors.

Example 24. the display system of example 1, wherein the operations further comprise: such that the light pattern is continuously scanned over an axis between the first portion of the eye and the second portion of the eye.

Example 25. the display system of example 24, wherein the axis is a horizontal axis of the eye such that the first portion is a leftmost portion or a rightmost portion of the eye and the second portion is the other of the leftmost portion or the rightmost portion of the eye.

Example 26. the display system of example 1, wherein determining the eye pose comprises:

communicating an application machine learning model via calculating a forward direction of the light intensity pattern, wherein an output of the machine learning model is indicative of an eye pose.

The display system of example 1, wherein determining the eye pose comprises:

accessing information identifying stored light intensity patterns associated with respective eye gestures;

comparing the obtained light intensity pattern with the stored light intensity pattern; and identifying the eye pose based on the comparison.

Example 28. the display system of example 26, wherein the light detector is a photodiode, wherein comparing the obtained light intensity pattern to the stored light intensity pattern is based on comparing locations of peaks and/or troughs of current, and wherein the locations are indicative of locations of the optical pattern on the eye.

Example 29. the display system of example 1, wherein the operations further comprise:

determining an interpupillary distance of the user;

determining a scanning distance to scan the light pattern on the eye based on the determined interpupillary distance; and

Scanning the light pattern over the eye at the scan distance.

Example 30. the display system of example 1, wherein the operations further comprise detecting one or both of an iris and a pupil of the eye based on the light intensity pattern.

Example 31 the display system of example 30, wherein detecting one or both of the iris and the pupil of the eye comprises: determining a size of one or both of the iris and the pupil of the eye.

Example 32 the display system of example 30, wherein detecting one or both of the iris and the pupil of the eye comprises: determining a location of one or both of the iris and the pupil of the eye.

Example 33. the display system of example 1, wherein the operations further comprise: determining a saccadic velocity of the eye.

Example 34 the display system of example 1, further comprising: a waveguide comprising out-coupling optical elements configured to output light to the user's eye to form the virtual content.

Example 35 the display system of example 29, wherein the waveguide is one waveguide in a stack of waveguides, wherein some waveguides in the stack have outcoupling optical elements configured to output light having a different amount of wavelength divergence than outcoupling optical elements of other waveguides in the stack, wherein the different amount of wave front divergence corresponds to a different depth plane.

An example 36. a method implemented by a display system of one or more processors, the display system configured to present virtual content to a user based at least in part on eye gestures of eyes of the user, wherein the method comprises:

adjusting a position of a light pattern directed onto the eye such that the light pattern moves over the eye;

obtaining a plurality of light intensity patterns representing photodetector signals at different times, the photodetector signals obtained from respective photodetectors during adjustment of the position of the light patterns; and

determining the eye pose of the eye based on the light intensity pattern, the eye pose representing an orientation of the eye.

Example 37. the method of example 36, wherein adjusting the position of the light pattern comprises: moving a movable mirror to cause the light pattern to move along an axis from a first portion of the eye to a second portion of the eye.

Example 38 the method of example 37, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light beams.

Example 39. the method of example 37, wherein moving the movable reflector includes rotating a micro-electromechanical systems (MEMS) mirror on which the diffraction grating is located.

Example 40. the method of example 37, wherein the first portion represents a terminal end of the iris and the second portion represents an opposite terminal end of the iris along the axis.

Example 41. the method of example 37, wherein the axis is a horizontal axis.

Example 42. the method of example 36, wherein the light pattern extends along a vertical axis from a lower portion of the eye to an upper portion of the eye.

Example 43 the method of example 42, wherein the light pattern includes two portions, each portion extending along a vertical axis, and wherein the two portions extend in opposite directions along a horizontal direction to form a V-shape.

The method of example 36, wherein determining the eye pose comprises:

applying a machine learning model via calculating a forward pass of the light intensity pattern, wherein an output of the machine learning model is indicative of an eye pose.

Example 45. the method of example 36, wherein determining the eye pose comprises:

comparing the obtained light intensity pattern with the stored light intensity pattern; and

identifying the eye pose based on the comparison.

Example 46. the method of example 45, wherein comparing the obtained light intensity pattern with the stored light intensity pattern is based on comparing locations of peaks and/or valleys in the light intensity pattern.

An example 47. a non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

adjusting a position of a light pattern directed onto a user's eye such that the light pattern moves on the eye;

Example 48 the computer storage medium of example 47, wherein the operations further comprise:

such that the light pattern is projected to the eye via a reflector having a diffraction grating.

Example 49. the computer storage medium of example 48, wherein the orientation of the diffraction grating is adjusted such that the light pattern moves from a first portion of the eye to a second portion of the eye.

Example 50. the computer storage medium of example 49, wherein the first portion represents a distal end of the iris and the second portion represents an opposite distal end of the iris.

Example 51. the computer storage medium of example 47, wherein the light pattern extends along a vertical axis from a lower portion of the eye to an upper portion of the eye.

Example 52 the computer storage medium of example 51, wherein the light portion comprises two portions, each portion extending along a vertical axis over the eye, and wherein the two portions extend along a horizontal axis in opposite directions.

Example 53 the computer storage medium of example 47, wherein adjusting the orientation of the diffraction grating comprises: controlling rotation of a micro-electro-mechanical system (MEMS) mirror on which the diffraction grating is located.

Example 54 the computer storage medium of example 47, wherein determining the eye pose comprises:

Example 55 the computer storage medium of example 47, wherein determining the eye pose comprises:

identifying the eye pose based on the comparison.

Example 56 the computer storage medium of example 55, wherein comparing the obtained light intensity pattern to the stored light intensity pattern is based on comparing locations of peaks and/or valleys.

Example 57 a display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

one or more processors configured to perform operations comprising:

obtaining respective light intensity patterns via the photodetectors, wherein the light intensity patterns represent photodetector signals at different times, the photodetector signals obtained during scanning of the reflected light on the eye; and

based on the light intensity pattern, one or both of a size and a position of a physiological feature of the eye is determined.

Example 58. the display system of example 57, wherein the physiological characteristic is a pupil of the eye.

Example 59. the display system of example 58, wherein the operations further comprise:

determining a first interface between an iris and a pupil of the eye based on the light intensity pattern.

Example 60. the display system of example 59, wherein determining the first interface is based on a location of a peak and/or a valley in the light intensity pattern.

Example 61. the display system of example 59, wherein the operations further comprise:

determining a second interface between an iris and a pupil of the eye based on the light intensity pattern.

Example 62. the display system of example 61, wherein the size of the pupil is determined based on the first interface and the second interface.

Example 63. the display system of example 61, wherein the physiological characteristic is the pupil, and wherein the position of the pupil is determined based on a center of the pupil, the center identified based on the first interface and the second interface.

Example 64. the display system of example 57, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

Example 65 a method implemented by a display system of one or more processors configured to present virtual content to a user based at least in part on eye gestures of eyes of the user, wherein the method comprises:

Determining a size and/or location of a physiological feature of the eye based on the light intensity pattern.

Example 66. the method of example 65, wherein the physiological characteristic is a pupil of the eye.

Example 67. the method of example 66, further comprising:

Example 68. the method of example 67, wherein determining the first interface is based on a location of a peak and/or a valley in the light intensity pattern.

Example 69. the method of example 68, further comprising:

Example 70. the method of example 69, wherein the physiological characteristic is the pupil, and wherein a size of the pupil is based on the first interface and the second interface.

Example 71. the method of example 69, wherein the physiological feature is the pupil, and wherein the location of the pupil is based on a center of the pupil, the center identified based on the first interface and the second interface.

Example 72. the method of example 65, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

Example 73. a non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

Example 74 the computer storage medium of example 73, wherein the operations further comprise:

Example 75. the computer storage medium of example 74, wherein determining the interface is based on locations of peaks and/or valleys in the light intensity pattern.

Example 76 the computer storage medium of example 74, wherein the operations further comprise:

determining a second interface between the iris and the pupil of the eye based on the light intensity pattern.

Example 77 the computer storage medium of example 76, wherein the physiological characteristic is the pupil, and wherein a size of the pupil is determined based on the first interface and the second interface.

Example 78 the computer storage medium of example 76, wherein the physiological characteristic is the pupil, and wherein the location of the pupil is based on a center of the pupil, the center identified based on the first interface and the second interface.

Example 79. the computer storage medium of example 73, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

An example 80. a display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

one or more processors configured to perform operations comprising:

determining a rotation speed of the eye based on the light intensity pattern.

Example 81. the display system of example 80, wherein determining the rotational speed of the eye comprises: determining a saccadic velocity of the eye.

Example 82. the display system of example 81, wherein the operations further comprise: predicting a pose of the eye based on the saccade velocity.

An example 83. a method implemented by a display system of one or more processors configured to present virtual content to a user based at least in part on eye gestures of eyes of the user, wherein the method comprises:

determining a rotation speed of the eye based on the light intensity pattern.

An example 84. a non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

adjusting a position of a light pattern directed onto a user's eye such that the light pattern moves over the eye;

determining a rotation speed of the eye based on the light intensity pattern.

Drawings

Fig. 1 shows a view of an Augmented Reality (AR) scene viewed by a user through an AR device.

Fig. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.

Fig. 3A to 3C show the relationship between the radius of curvature and the focal radius.

Fig. 4A shows a representation of the accommodation-convergence reaction of the human visual system.

Fig. 4B shows examples of different accommodation states and convergence states of a pair of eyes of a user.

Fig. 4C illustrates an example of a representation of a top view of content viewed by a user via a display system.

FIG. 4D illustrates another example of a representation of a top view of content viewed by a user via a display system.

FIG. 5 illustrates aspects of a method for simulating a three-dimensional image by modifying wavefront divergence.

Fig. 6 shows an example of a waveguide stack for outputting image information to a user.

Fig. 7 shows an example of an outgoing light beam output by a waveguide.

Fig. 8 illustrates an example of a stacked waveguide assembly, wherein each depth plane includes images formed using a plurality of different component colors.

Fig. 9A illustrates a cross-sectional side view of an example of stacked waveguide sets each including an incoupling optical element.

Fig. 9B illustrates a perspective view of an example of the plurality of stacked waveguides of fig. 9A.

Fig. 9C illustrates a top plan view of an example of the multiple stacked waveguides of fig. 9A and 9B.

Fig. 9D illustrates an example of a wearable display system.

FIG. 10A illustrates a plan view of a system and technique for determining a user's eye pose.

FIG. 10B illustrates an example reflected light intensity pattern associated with scanning a user's eye with a light pattern.

FIG. 11 illustrates example positioning of a light detector within a display system for determining eye pose.

FIG. 12 illustrates an example flow diagram of a process for determining an eye pose of a user's eyes.

Fig. 13A-13C illustrate examples of light patterns being projected and scanned on a user's eye.

Fig. 14A-14C illustrate another example of a light pattern being projected onto and scanned over a user's eye, where the user's eye is in a different pose than the eye shown in fig. 13A-13C.

Fig. 15 shows an example of a light pattern for scanning over a user's eye.

Fig. 16 shows another example of a light pattern for scanning over a user's eye.

Fig. 17A to 17C illustrate the use of two light sources for generating a light pattern for determining the posture of the user's eyes.

Fig. 18A illustrates an example flow diagram of a process for determining physiological information associated with an eye.

18B-D illustrate examples of determining size and/or location information associated with one or more physiological characteristics.

Detailed Description

The AR and/or VR system may display the virtual content to a user or viewer. For example, the content may be displayed on a head-mounted display, e.g., as part of glasses, which project image information to the user's eyes. Additionally, where the system is an AR system, the display may also transmit light from the surrounding environment to the user's eyes to allow viewing of the surrounding environment. As used herein, it will be understood that a "head mounted" or "head wearable" display is a display that may be mounted on the head of a viewer or user. Such a display may be understood to form part of a display system.

To provide visually realistic virtual content, it is advantageous for the display system to accurately track (e.g., monitor) the user's eyes. For example, accurate determinations regarding the orientation of each eye (referred to herein as eye pose) may enhance the realism of the rendered virtual content. In practice, a virtual scene (e.g., the augmented reality scene 10 shown in fig. 1) may be rendered by a display system based on the user's eyes being assigned as "rendering cameras" for the scene. For example, the center of the user's eyes may be assigned as the rendering camera. Thus, the location of the virtual content within the virtual scene may be related to the center of the user's eyes along with the gaze direction and vergence of their eyes. As the user moves his/her eyes, for example, to view virtual content or real-world content, the display system may adjust the virtual content accordingly. Thus, enhanced techniques for tracking the eyes of a user may substantially enhance the functionality of such display systems and provide a better viewing experience for the user.

Tracking the user's eyes may include determining a vergence, a gaze direction, a respective center of the user's eyes, and the like. At least some of these determinations may be implemented based on recognition of respective eye gestures of the user's eyes. For example, based on the orientation of the eye, the display system may determine an axis (e.g., an optical axis and/or a visual axis) extending from the eye. The axis may represent the gaze direction of the user's eyes. Using eye gestures for both eyes of the user, the display system may identify a location in three-dimensional space that the user's eyes are approaching.

It should be appreciated that gaze direction tracking may be used to determine virtual content displayed to a user; for example, virtual content related to the real world may be adjusted to provide the correct correspondence to the real world by tracking where the user is looking. In addition, in display systems that provide virtual content on different depth planes, the point that the user's eye is approaching can be utilized to determine the appropriate depth plane on which to display the virtual content.

Some prior art techniques for determining eye pose are limited in tracking speed, such that updates to eye pose may be constrained. This may cause undesirable delays or jitter when updating the content to be displayed. Furthermore, some prior art techniques for determining eye pose have high power requirements. Moreover, some prior art techniques require hardware and/or optical structures, which undesirably increase the complexity of the manufacturing process for forming display systems having such hardware or systems.

For example, eye tracking systems that utilize cameras and analysis of captured images to determine eye pose may be undesirably limited in tracking speed, may utilize a large amount of power, and may require complex and accurate camera and light source arrangements. Such camera-based systems may use a plurality of Light Emitting Diodes (LEDs) to project light at the eyes of the user. The LEDs may be positioned on the display system such that light from the LEDs reflects off a particular portion of the user's eye (e.g., the pupil). A camera may be positioned on the display system to image the eye and determine the location of the reflected light. As the user moves his/her eyes (e.g., changes eye pose), the image and position of the reflected light may similarly change. Based on analysis of the captured image of the eye, the display system may determine the eye pose.

The example techniques described above may allow for accurate determination of a user's eye pose. However, they present certain technical challenges. For example, the eye tracking speed may be limited by the rate at which the camera can capture and process images (e.g., at a rate of 60 Hz). As an example, constraints on tracking speed may limit the extent to which the currently determined eye pose may be relied upon. As an example, during certain rapid movements of the eyes (e.g., saccadic movements), the tracking velocity may lag behind the eye's movement such that the displayed content may not match, e.g., the user's gaze direction or vergence of the eyes. Furthermore, the limitation on the tracking speed may introduce some visual artifacts. For example, jitter may be introduced when rendering virtual content. Jitter may be caused, at least in part, by the determined eye pose being periodically inaccurate when displaying virtual content.

In addition to the challenges described above in presenting virtual content, existing camera imaging-based techniques for pose determination may have electrical and mechanical challenges. With respect to electrical constraints, power consumption can be high. In practice, multiple light sources are required to be driven and a camera (e.g., an infrared camera) is required to capture an image of the user's eye. Due to the complexity of the image information, processing aspects are added that require the ability to analyze each image. The power requirement is in addition to the power required to render the virtual content. As a result, the portability and battery life of displays using such camera imaging based techniques may be undesirably low. With respect to mechanical constraints, there may be complications associated with positioning the LEDs and camera. For example, it may be desirable to align the LEDs so that they project light to certain locations of the user's eyes. As another example, it may be desirable to aim the camera so that it obtains an image in which all or part of the LEDs are visible in each image of the user's eyes regardless of eye pose.

As discussed herein, various embodiments provide enhanced techniques and systems for tracking a user's eyes and advantageously address one or more of the above-described technical problems. For example, eye tracking techniques may track at extremely high speeds (e.g., 1kHz, 10kHz, etc.). In some embodiments, the eye tracking techniques described herein may use only one light source per eye. For example, a diode (e.g., a vertical cavity surface emitting laser diode) or other light emitting device may be used for one or more eyes. In contrast to the requirements of a camera, the techniques described herein may use a threshold number of photodetectors (e.g., photodiodes, phototransistors, etc.) to detect the amount of reflected incident light and process the light intensity signal provided by the reflected light rather than imaging the eye. Thus, the power requirements may be substantially less (e.g., an order of magnitude or less). Furthermore, the positioning of the diodes and light detectors may be substantially simpler compared to the techniques described above, since gesture detection is based on a pattern of detected light intensity over time, rather than image analysis of the precise location of reflected light in the captured image.

To determine an eye pose for a user's eye, the techniques described herein may scan light, such as a particular light pattern, over the user's eye. As an example, a light pattern may be projected such that it scans (e.g., sweeps) over the user's eyes along a horizontal axis (which may be understood as extending through the center of the user's left and right eyes). In some embodiments, the light pattern may include a light ray or a plurality of spots forming lines extending perpendicularly at an angle of less than 90 ° above the user's eye. For example, the light pattern may extend vertically from a lower portion of the user's eye to an upper portion of the eye. The light pattern may then be moved along a horizontal axis such that it scans the width of the user's eye or a portion thereof. In some other embodiments, scanning the light pattern over the user's eye may involve moving the light pattern along a vertical axis over the height of the user's eye.

As will be described below, a movable reflector, such as a micro-electromechanical system (MEMS) mirror, may be used to cause the light pattern to be scanned. In some embodiments, the movable reflector may have a diffraction grating. Light from the light source may be directed to a diffraction grating (also referred to as a diffraction grating) such that a particular light pattern is created. The movable reflector may then be moved (rotated) about one or more axes such that the light pattern is scanned over the user's eye, as described above. In some embodiments, the movable reflector may be movable about a single axis. For example, in some such embodiments, the movable reflector may be a one-dimensional MEMS mirror, and the light pattern employed may vary spatially in at least one other dimension.

In some embodiments, the light pattern may be a "V" pattern. Examples of such light patterns are shown in fig. 10A-10B and described in more detail herein. The example pattern may include two portions that each extend from a lower portion of the user's eye to an upper portion along a vertical axis. However, each portion may extend over a different horizontal portion of the user's eye. For example, each portion may be an angled line extending in opposite horizontal directions from the same lower portion (e.g., two portions of a light pattern may form a "V").

As another example, a first portion of the light pattern may be scanned horizontally over the user's eye and then a second portion of the light pattern may also be scanned horizontally over the user's eye. In some embodiments, the first portion and the second portion may be two legs of a "V" pattern. For example, the first portion may be the right leg of a "V" shape, such that the upper end guides the lower end of the first portion on the eye (referred to herein as the "a" portion). A second portion of the light pattern (the left leg of the "V") may similarly be scanned horizontally over the user's eye such that the lower end leads the upper end of the second portion (referred to herein as the "β" portion). It will be appreciated that the first and second portions extend in different directions. In some embodiments, the "V" shape may be inverted to assume a "Λ" shape.

It should be understood that other light patterns may be used and fall within the scope of the disclosure herein. In some embodiments, the reflector may have a plurality of diffraction gratings that provide different light patterns depending on the diffraction grating upon which light from the light source is incident.

The user's eye may reflect the light pattern directed thereon by the movable reflector. To determine eye pose, the display system may use the light detector to measure information associated with the reflection (e.g., the intensity of light incident on the light detector). As an example, a display system may use a photodiode to convert received light into a corresponding current. These photodiodes may preferably be at different positions on the display system relative to the eye. Fig. 11 shows an example orientation of a photodiode. As the light pattern is scanned across the eye, each photodiode located at a different location receives a different reflected light from the eye and converts the different received light into a different pattern or current versus time plot. For ease of reference, the intensity of reflected light detected by the light detector at different points in time, when the light pattern is scanned over the eye, is referred to herein as a light intensity pattern. In some embodiments, the light intensity pattern may correspond to a pattern defined or derived from the current at different points in time or at different locations during the scan, where the reflected light is detected by a light sensor (e.g., a photodiode) that converts the light into the current. In some embodiments, each of one or more of the light detectors in the display system may be electrically coupled to generate a voltage indicative of the intensity of light incident on the detector of the respective light. In at least some of these embodiments, one or more photodiodes, phototransistors, and/or photoresistors may be used as photodetectors in a display system.

Thus, for each scan of the light pattern over the user's eye, there may be a large number of generated light intensity patterns, each detected by a different light detector. As discussed herein, the display system may use these light intensity patterns to determine an eye pose for the user's eyes. It should be appreciated that different portions of the user's eye may cause light to be reflected differently (due to asymmetry in the shape of the eye and/or differences in the composition of different portions of the eye). Thus, for a given orientation of the user's eye, the set of resulting light intensity patterns obtained from the different light detectors may be substantially unique; that is, a given eye pose may have a unique set of "signatures" defined by a set of light intensity patterns. Using a threshold number of these light intensity patterns, the display system may determine the eye pose by determining the eye pose associated with the light intensity pattern.

In some embodiments, the light intensity pattern may include light having different characteristics so as to effectively provide a plurality of different, differentiated signals each time passing through the eye. For example, the light of the pattern may comprise a plurality of lines of light, each line of light having, for example, a different wavelength or a different polarization. In some embodiments, each row of light may include multiple beams of light, for example, a row of light formed from a converging beam of light, a row of light formed from a collimated beam of light, and/or a row of light formed from a diverging beam of light.

To determine eye pose, in some embodiments, the display system may access stored information that may be used to correlate light intensity patterns to eye pose. For example, a machine learning model may be used to determine eye pose based on input of light intensity patterns. Alternatively, the stored information may represent light intensity patterns known to be associated with certain eye gestures. In some embodiments, the display system may determine the eye pose based on analyzing the light intensity pattern. For example, the display system may use peaks, valleys, curvatures, etc. as represented in the light intensity pattern to determine the eye pose. In this example, the display system may correlate between different light intensity patterns to determine eye pose.

Accordingly, various embodiments provide improvements and address technical challenges associated with eye tracking or eye pose determination in display systems. As described above, the techniques described herein may allow for technical efficiencies. For example, the frequency or rate of determining eye pose may be increased. This increased frequency may allow for improved visual fidelity, realism, and viewing comfort when using the display system. Furthermore, power requirements and mechanical alignment complexity may be reduced.

Advantageously, it should be understood that the light intensity pattern contains additional information that may be used for other purposes. For example, as discussed herein, different portions of the eye (e.g., the sclera, iris, and pupil) have different reflectivities, which provide different levels of reflected light intensity. These different reflex tendencies may be used to determine the location and/or size of physiological features of the eye (e.g., the iris and/or pupil). Additionally, in some embodiments, the display system may determine the velocity of eye movement, e.g., the velocity of a saccade, which may be useful for predicting the position of the eye after a saccade. In some embodiments, the prediction may be used as a check on the detected eye pose.

Reference will now be made to the drawings wherein like reference numerals refer to like parts throughout. The drawings are schematic and are not necessarily drawn to scale unless specifically indicated otherwise.

Example display System

Fig. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. It will be appreciated that the eyes of the user are spaced apart from each other, and that when viewing a real object in space, each eye has a slightly different view of the object, and an image of the object may be formed at a different location on the retina of each eye. This may be referred to as binocular disparity and may be used by the human visual system to provide a sense of depth. Conventional display systems simulate binocular disparity by presenting two distinct images 190, 200 with slightly different views of the same virtual object, one for each

eye

210, 220, corresponding to a virtual object view where each eye sees the virtual object as a real object at the desired depth. These images provide binocular cues that the user's visual system can interpret to infer a sense of depth.

With continued reference to fig. 2, the images 190, 200 are spaced from the

eyes

210, 220 along the z-axis by a distance 230. The z-axis is parallel to the optical axis of a viewer whose eyes fixate on objects at optical infinity directly in front of the viewer. The images 190, 200 are flat and located at a fixed distance from the

eyes

210, 220. Based on the slightly different virtual object views in the images presented to the

eyes

210, 220, respectively, the eyes can naturally rotate so that the image of the object falls on the corresponding point on the retina of each eye, thereby maintaining single binocular vision. This rotation may cause the line of sight of each

eye

210, 220 to converge on a point in space where the virtual object is perceptually presented. Thus, providing a three-dimensional image typically involves providing binocular cues that can manipulate the vergence of the user's

eyes

210, 220, and the human visual system interprets the cues to provide a sense of depth.

However, creating a realistic and comfortable depth perception is challenging. It should be understood that light from objects at different distances relative to the eye have wavefronts with different amounts of divergence. Fig. 3A to 3C show the relationship between the distance and the light ray divergence. The distances between the object and the eye 210 are represented by R1, R2, and R3 in order of decreasing distance. As shown in fig. 3A to 3C, as the distance to the object decreases, the light rays become more divergent. Conversely, as the distance increases, the light rays become more collimated. In other words, the light field produced by a point (object or portion of an object) will be considered to have a spherical wavefront curvature that is a function of the distance of the point from the user's eye. As the distance between the object and the eye 210 decreases, the curvature increases. Although only a single eye 210 is shown for clarity in fig. 3A-3C and other figures herein, the discussion regarding eye 210 may be applied to both

eyes

210 and 220 of the viewer.

With continued reference to fig. 3A-3C, light from an object at which the viewer's eyes are gazing may have varying degrees of wavefront divergence. Due to the different amounts of wavefront divergence, the light may be focused differently by the lens of the eye, which in turn may require the lens to assume different shapes to form a focused image on the retina of the eye. In the case where a focused image is not formed on the retina, the resulting retinal blur acts as an accommodation cue that causes a change in the shape of the eye's lens until a focused image is formed on the retina. For example, the accommodative cues may trigger relaxation or contraction of the ciliary muscle around the lens of the eye to accommodate the force applied to the zonules holding the lens, thereby changing the shape of the lens of the eye until the retinal blur of the object of fixation is eliminated or minimized, thereby forming a focused image of the object of fixation on the retina (e.g., the fovea) of the eye. The process by which the eye lens changes shape may be referred to as accommodation, and the shape of the eye lens required to form a focused image of a fixation object on the retina (e.g., fovea) of the eye may be referred to as an accommodation state.

Referring now to fig. 4A, a representation of the accommodation-vergence response of the human visual system is shown. The eye moving to fixate on the object causes the eye to receive light from the object, wherein the light forms an image on each retina of the eye. The presence of retinal blur in the image formed on the retina can provide accommodative cues and the relative position of the image on the retina can provide vergence cues. The accommodation cues cause accommodation to occur such that the eye lenses each assume a particular state of accommodation that forms a focused image of the object on the retina (e.g., the fovea) of the eye. On the other hand, the vergence cues cause vergence motion (eye rotation) to occur such that the image formed on each retina of each eye is at the corresponding retinal point that maintains a single binocular vision. At these locations, the eyes may be considered to have assumed a particular convergence state. With continued reference to fig. 4A, accommodation can be understood as the process by which the eyes achieve a particular state of accommodation, and convergence can be understood as the process by which the eyes achieve a particular state of convergence. As shown in fig. 4A, if the user gazes at another object, the accommodation and convergence states of the eyes may be changed. For example, if the user gazes at a new object at a different depth on the z-axis, the adjustment state may change.

Without being limited by theory, it is believed that a viewer of the object may perceive the object as "three-dimensional" due to a combination of convergence and accommodation. As described above, the vergence movement of the two eyes relative to each other (e.g., rotating the pupils close to or away from each other to converge the visual lines of the eyes to gaze at the subject) is closely related to the accommodation of the eye lens. Under normal circumstances, changing the shape of the lens of the eye to change focus from one object to another at a different distance will automatically result in a change in vergence match to the same distance, according to a relationship known as the "accommodation-vergence reflex". Also, under normal conditions, a change in vergence will cause a change in the lens shape fit.

Referring now to fig. 4B, examples of different states of accommodation and vergence of the eyes are shown. The pair of eyes 222a fixate on an object at optical infinity, and the pair of eyes 222b fixate on an object 221 at less than optical infinity. It is to be noted that the convergence state of each pair of eyes is different, the pair of eyes 222a gaze right ahead, and the pair of eyes 222 converge on the object 221. The accommodation state of the eyes forming each pair of eyes 222a and 222b may also be different, as indicated by the different shapes of lenses 210a, 220 a.

Undesirably, many users of conventional "3-D" display systems find these conventional systems uncomfortable or unable to perceive a sense of depth at all because of the mismatch between the accommodation and vergence states in these displays. As described above, many stereoscopic or "3-D" display systems display a scene by providing slightly different images to each eye. Such systems are uncomfortable for many viewers because one of the factors is that they only provide different scene presentations and cause a change in the vergence state of the eyes, without a corresponding change in the accommodative state of the eyes. Instead, the image is shown by a display located at a fixed distance from the eye, so that the eye views all image information in a single state of accommodation. This arrangement works against the "accommodation-vergence reflex" relationship by causing a vergence state change without a matching accommodation state change. This mismatch is believed to cause viewer discomfort. A display system that provides a better match between accommodation and convergence can create a more realistic and comfortable three-dimensional image simulation.

Without being limited by theory, it is believed that the human eye can typically interpret a limited number of depth planes to provide depth perception. Thus, highly reliable perceptual depth simulations may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. In some embodiments, the different presentations can provide a vergence cue and a matching accommodative cue, thereby providing a physiologically correct accommodative-vergence match.

With continued reference to fig. 4B, two depth planes 240 are shown, corresponding to different distances in space from the

eyes

210, 220. For a given depth plane 240, the vergence cue can be provided by displaying appropriate different viewing angle images for each

eye

210, 220. Further, for a given depth plane 240, the light forming the image provided to each

eye

210, 220 may have a wavefront divergence corresponding to the light field produced by the point at that distance of that depth plane 240.

In the illustrated embodiment, the distance along the z-axis of the depth plane 240 containing the point 221 is 1 m. As used herein, the distance or depth along the z-axis may be measured with a zero point located at the exit pupil of the user's eye. Thus, the depth plane 240 at a depth of 1m corresponds to a distance of 1m from the exit pupil of the user's eye on the visual axis of these eyes, where the eye points to optical infinity. As an approximation, the depth or distance along the z-axis can be measured as: the distance from the display in front of the user's eye (e.g., from the waveguide surface) plus the distance value between the device and the exit pupil of the user's eye. This value may be referred to as the exit pupil spacing and is the distance between the user's eye exit pupil and the display worn by the user in front of the user's eye. In practice, the exit pupil interval value may be a standardized value that is typically used for all viewers. For example, the exit pupil spacing is 20mm, then a depth plane at a depth of 1m may be located at a distance of 980mm in front of the display.

Referring now to fig. 4C and 4D, examples of matched vergence distance and mismatched vergence distance are shown, respectively. As shown in fig. 4C, the display system may provide an image of the virtual object to each

eye

210, 220. The images may cause the

eyes

210, 220 to assume a vergence state in which the eyes converge on point 15 on the depth plane 240. Additionally, the image may be formed from light having a wavefront curvature corresponding to the real object at the depth plane 240. Thus, the

eyes

210, 220 assume an accommodation state in which the image is focused on the retinas of these eyes. Thus, the user may perceive that the virtual object is located at point 15 on the depth plane 240.

It should be appreciated that each of the accommodation and vergence states of the

eyes

210, 220 are associated with a particular distance on the z-axis. For example, objects at a particular distance from the

eyes

210, 220 cause the eyes to assume a particular state of accommodation based on the distance of the object. The distance associated with a particular adjustment state may be referred to as an adjustment distance A_d. Similarly, there is an eye at a particular convergence stateOr a specific convergence distance V associated with positions opposite to each other _d. In the case where the adjustment distance and the convergence distance match, the relationship between the adjustment and the convergence can be considered to be physiologically correct. This is considered to be the most comfortable situation for the viewer.

However, in a stereoscopic display, the adjustment distance and the convergence distance may not always match. For example, as shown in fig. 4D, the image displayed to the

eye

210, 220 may be displayed by wavefront divergence corresponding to the depth plane 240, and the

eye

210, 220 may assume a particular state of accommodation in which focus is on a point 15a, 15b on the depth plane. However, the images displayed to the

eyes

210, 220 may provide a vergence cue to cause the

eyes

210, 220 to converge on a point 15 that is not located on the depth plane 240. Thus, in some embodiments, the accommodation distance corresponds to the distance from the exit pupils of the

eyes

210, 220 to the depth plane 240, while the vergence distance corresponds to the greater distance from the exit pupils of the

eyes

210, 220 to the point 15. The adjustment distance is different from the convergence distance. Thus, there is a vergence mismatch of accommodation. This mismatch is considered undesirable and may cause discomfort to the user. It should be understood that mismatch corresponds to distance (e.g., V) _d-A_d) And may be characterized using diopters.

In some embodiments, it will be understood that the reference points outside the exit pupils of the

eyes

210, 220 can be used to determine the distance for determining the accommodation-vergence mismatch, so long as the same reference points are used for the accommodation and vergence distances. For example, distances may be measured from the cornea to a depth plane, from the retina to a depth plane, from an eyepiece (e.g., a waveguide of a display device) to a depth plane, and so forth.

Without being limited by theory, it is believed that the user can still perceive accommodation-vergence mismatch as physiologically correct up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters without the mismatch itself causing significant discomfort. In some embodiments, the display systems disclosed herein (e.g., display system 250 of fig. 6) present images to a viewer with an accommodative-vergence mismatch of about 0.5 diopters or less. In some other embodiments, the vergence mismatch of the images provided by the display system is about 0.33 diopters or less. In still other embodiments, the vergence mismatch of the images provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

FIG. 5 illustrates aspects of a method for simulating a three-dimensional image by modifying wavefront divergence. The display system includes a waveguide 270, the waveguide 270 configured to receive light 770 encoded with image information and output the light to a user's eye 210. Waveguide 270 may output light 650 having a defined amount of wavefront divergence corresponding to the wavefront divergence of the optical field produced by the point on the desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on the depth plane. In addition, it will be shown that image information from a similar waveguide may be provided to the other eye of the user.

In some embodiments, a single waveguide may be configured to output light having a set amount of wavefront divergence corresponding to a single depth plane or a limited number of depth planes, and/or a waveguide may be configured to output light of a limited wavelength range. Thus, in some embodiments, multiple waveguides or waveguide stacks may be utilized to provide different amounts of wavefront divergence for different depth planes and/or to output light having different wavelength ranges. As used herein, it will be understood that the depth plane may be planar or may follow the contour of a curved surface.

Fig. 6 shows an example of a waveguide stack for outputting image information to a user. The display system 250 includes a stack of waveguides or a stacked waveguide assembly 260 that can be used to provide three-dimensional perception to the eye/brain using a plurality of

waveguides

270, 280, 290, 300, 310. It will be understood that in some embodiments, the display system 250 may be considered a light field display. Further, the waveguide assembly 260 may also be referred to as an eyepiece.

In some embodiments, the display system 250 can be configured to provide a substantially continuous vergence cue and a plurality of discontinuous accommodative cues. The vergence cues may be provided by displaying a different image to each eye of the user, and the accommodation cues may be provided by outputting light that forms an image with a selectable discrete amount of wavefront divergence. In other words, the display system 250 may be configured to output light having a variable wavefront divergence level. In some embodiments, each discrete level of wavefront divergence corresponds to a particular depth plane, and may be provided by a particular one of the

waveguides

270, 280, 290, 300, 310.

With continued reference to fig. 6, the waveguide assembly 260 may also include a plurality of

features

320, 330, 340, 350 located between the waveguides. In some embodiments, the

features

320, 330, 340, 350 may be one or more lenses. The

waveguides

270, 280, 290, 300, 310 and/or the plurality of

lenses

320, 330, 340, 350 may be configured to transmit image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane.

Image injection devices

360, 370, 380, 390, 400 may be used as light sources for the waveguides and may be used to inject image information into

waveguides

270, 280, 290, 300, 310, each of which may be configured to distribute incident light on each respective waveguide for output toward eye 210, as described herein. Light is emitted from the output surfaces 410, 420, 430, 440, 450 of the

image injection devices

360, 370, 380, 390, 400 and injected into the corresponding input surfaces 460, 470, 480, 490, 500 of the

waveguides

270, 280, 290, 300, 310. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 can be an edge of the corresponding waveguide, or can be a portion of a major surface of the corresponding waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single beam (e.g., a collimated beam) may be injected into each waveguide to output a whole cloned collimated beam field, which is directed toward the eye 210 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, a single one of the

image injection devices

360, 370, 380, 390, 400 may be associated with and inject light into multiple ones (three) of the

waveguides

270, 280, 290, 300, 310.

In some embodiments, the

image injection devices

360, 370, 380, 390, 400 are separate displays, each of which produces image information for injection into a

corresponding waveguide

270, 280, 290, 300, 310, respectively. In some other embodiments, the

image injection devices

360, 370, 380, 390, 400 are outputs of a single multiplexed display that can, for example, pipe image information to each of the

image injection devices

360, 370, 380, 390, 400 via one or more optical conduits (e.g., fiber optic cables). It will be appreciated that the image information provided by the

image injection devices

360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different component colors as discussed herein).

In some embodiments, the light injected into the

waveguides

270, 280, 290, 300, 310 is encoded with image information and provided by a light projector system 1010, as discussed further herein. In some embodiments, the light projector system 1010 may include one or more arrays of emission pixels. It should be understood that the array of emissive pixels may each include a plurality of emissive pixels, which may be configured to emit light of different intensities and colors. It should be understood that the

image injection devices

360, 370, 380, 390, 400 are schematically shown and in some embodiments may represent different optical paths and locations in a common projection system configured to output light to an associated one of the

waveguides

270, 280, 290, 300, 310. In some embodiments, the waveguides of the waveguide assembly 260 can act as ideal lenses while relaying light injected into the waveguides to the user's eye. In this concept, the object may be a pixel array of the light projection system 1010 and the image may be an image on a depth plane.

The controller 560 controls operation of one or more of the stacked waveguide assemblies 260, including operation of the

image injection devices

360, 370, 380, 390, 400, the light projection system 540. In some embodiments, controller 560 is part of local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transitory medium) that adjusts the timing and provision of image information to the

waveguides

270, 280, 290, 300, 310 according to, for example, any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by a wired or wireless communication channel. In some embodiments, controller 560 may be part of processing module 140 or 150 (fig. 9D).

With continued reference to fig. 6, the

waveguides

270, 280, 290, 300, 310 may be configured to propagate light within each respective waveguide by Total Internal Reflection (TIR). The

waveguides

270, 280, 290, 300, 310 may each be planar or have another shape (e.g., curved) with top and bottom major surfaces and edges extending between the top and bottom major surfaces. In the illustrated configuration, the

waveguides

270, 280, 290, 300, 310 may each include outcoupling

optical elements

570, 580, 590, 600, 610 configured to extract light from the waveguides by redirecting light propagating within each respective waveguide outside the waveguide, thereby outputting image information to the eye 210. The extracted light may also be referred to as outcoupled light, and the outcoupled optical element may also be referred to as a light extraction optical element. The extracted light beam may be output by the waveguide at a location where light propagating within the waveguide strikes the light extraction optical element. The outcoupling

optical elements

570, 580, 590, 600, 610 may be, for example, gratings comprising diffractive optical features as discussed further herein. Although the out-coupling

optical elements

570, 580, 590, 600, 610 are shown disposed at the bottom major surface of the

waveguides

270, 280, 290, 300, 310 for ease of description and clarity of depiction, in some embodiments the out-coupling

optical elements

570, 580, 590, 600, 610 may be disposed at the top major surface and/or the bottom major surface, and/or may be disposed directly in the body of the

waveguides

270, 280, 290, 300, 310, as discussed further herein. In some embodiments, the out-coupling

optical elements

570, 580, 590, 600, 610 may be formed in a layer of material attached to a transparent substrate, thereby forming the

waveguides

270, 280, 290, 300, 310. In some other embodiments, the

waveguides

270, 280, 290, 300, 310 may be a monolithic piece of material, and the out-coupling

optical elements

570, 580, 590, 600, 610 may be formed on a surface and/or within the bulk of the material.

With continued reference to fig. 6, as discussed herein, each

waveguide

270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to deliver collimated light injected into such waveguide 270 to the eye 210. The collimated light may represent an optically infinite focal plane. The next uplink waveguide 280 may be configured to emit collimated light that is transmitted through a first lens 350 (e.g., a negative lens) before reaching the eye 210; such a first lens 350 may be configured to produce a slightly convex wavefront curvature such that the eye/brain interprets light from this next upstream waveguide 280 as coming from a first focal plane from optical infinity inward closer to the eye 210. Similarly, the third upstream waveguide 290 has its output light transmitted through both the first lens 350 and the second lens 340 before reaching the eye 210; the combined optical power of the first lens 350 and the second lens 340 can be configured to produce another wavefront curvature increment such that the eye/brain interprets light from the third waveguide 290 as light from a second focal plane further closer to the person from optical infinity inward than light from the next upward traveling waveguide 280.

The

other waveguide layers

300, 310 and

lenses

330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all lenses between it and the eye to obtain a total power representing the focal plane closest to the person. To compensate the lens stacks 320, 330, 340, 350 when viewing/interpreting light from the world 510 on the other side of the stacked waveguide assembly 260, a compensating lens layer 620 may be provided on top of the stack to compensate for the total power of the

underlying lens stacks

320, 330, 340, 350. This configuration provides as many sensing focal planes as there are waveguide/lens pairs available. Both the out-coupling optical elements of the waveguide and the focusing aspects of the lens may be static (i.e., not dynamic or electro-active). In some alternative embodiments, one or both of the out-coupling optical elements of the waveguide and the focusing aspects of the lens may be dynamic through the use of electro-active features.

In some embodiments, two or more of the

waveguides

270, 280, 290, 300, 310 may have the same associated depth plane. For example,

multiple waveguides

270, 280, 290, 300, 310 may be configured to output a set of images to the same depth plane, or multiple subsets of

waveguides

270, 280, 290, 300, 310 may be configured to output a set of images to the same multiple depth planes, one set for each depth plane. This may provide an advantage for forming tiled images to provide an extended field of view at those depth planes.

With continued reference to fig. 6, the out-coupling

optical elements

570, 580, 590, 600, 610 can be configured to both redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or degree of collimation for the particular depth plane associated with the waveguides. Thus, waveguides with different associated depth planes may have different configurations of out-coupling

optical elements

570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the out-coupling

optical elements

570, 580, 590, 600, 610 may be bulk features or surface features, which may be configured to output light at a particular angle. For example, the out-coupling

optical elements

570, 580, 590, 600, 610 may be volume holograms, surface holograms, and/or diffraction gratings. In some embodiments, the

features

320, 330, 340, 350 may not be lenses; rather, they may simply be spacers (e.g., cladding and/or structures for forming air gaps).

In some embodiments, the outcoupling

optical elements

570, 580, 590, 600, 610 are diffractive features that form a diffraction pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOEs have a sufficiently low diffraction efficiency such that only a portion of the light beam is deflected towards the eye 210 by each intersection point of the DOE, while the remainder continues through the waveguide via TIR. Thus, the light carrying the image information is split into a plurality of relevant exit beams that exit the waveguide at a plurality of locations, resulting in a fairly uniform exit pattern towards the eye 210 for this particular collimated beam bouncing around within the waveguide.

In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a polymer dispersed liquid crystal layer, wherein the droplets comprise a diffraction pattern in the host medium, and the refractive index of the droplets may be switched to substantially match the refractive index of the host material (in which case the pattern DOEs not significantly diffract incident light) or the droplets may be switched to a refractive index that is mismatched to the refractive index of the host medium (in which case the pattern actively diffracts incident light).

In some embodiments, a camera component 630 (e.g., a digital camera, including visible and infrared light cameras) may be provided to capture images of the eye 210 and/or tissue surrounding the eye 210, for example, to detect user input and/or to monitor a physiological state of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera component 630 may include an image capture device and a light source to project light (e.g., infrared light) to the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 630 may be attached to the frame 80 (fig. 9D) and may be in electrical communication with the processing module 140 and/or 150, and the processing module 140 and/or 150 may process image information from the camera assembly 630. In some embodiments, one camera component 630 may be used for each eye to monitor each eye separately.

Referring now to fig. 7, an example of an outgoing beam output by a waveguide is shown. One waveguide is shown, but it is understood that other waveguides in the waveguide assembly 260 (fig. 6) may function similarly, where the waveguide assembly 260 includes a plurality of waveguides. Light 640 is injected into waveguide 270 at input surface 460 of waveguide 270 and propagates within waveguide 270 by TIR. At the point where the light 640 strikes the DOE 570, a portion of the light exits the waveguide as an exit beam 650. The exit beams 650 are shown as being substantially parallel, but as discussed herein, they may also be redirected to propagate to the eye 210 at an angle (e.g., forming a diverging exit beam) that depends on the depth plane associated with the waveguide 270. It should be understood that a substantially parallel outgoing beam may be indicative of a waveguide having outcoupled optical elements that couple light out to form an image that appears to be disposed on a depth plane at a greater distance (e.g., optical infinity) from the eye 210. Other waveguides or other sets of out-coupling optical elements may output more divergent exit beam patterns that would require the eye 210 to accommodate closer distances to focus on the retina, and these beam patterns may be interpreted by the brain as light from distances closer to the eye 210 than optical infinity.

In some embodiments, a full color image may be formed at each depth plane by superimposing an image in each of the component colors (e.g., three or more component colors). Fig. 8 illustrates an example of a stacked waveguide assembly, wherein each depth plane includes images formed using a plurality of different component colors. The illustrated embodiment shows depth planes 240a through 240f, but more or less depths are also contemplated. Each depth plane may have associated therewith three or more component color images including: a first image of a first color G; a second image of a second color R; and a third image of a third color B. The different depth planes are indicated in the figure by the letters G, R and the different diopters (dpt) after B. By way of example only, the number following each of these letters represents diopter (1/m), or the reciprocal of the distance of the depth plane from the viewer, and each box in the figure represents a separate component color image. In some embodiments, the precise placement of the depth planes of different component colors may be varied in order to account for differences in the focus of the eye on light of different wavelengths. For example, different component color images for a given depth plane may be placed on the depth plane corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort, and/or may reduce chromatic aberrations.

In some embodiments, each component color of light may be output by a single dedicated waveguide, and thus, each depth plane may have multiple waveguides associated therewith. In such embodiments, each box in the figure including the letter G, R or B may be understood to represent a separate waveguide, and three waveguides may be provided for each depth plane, with three component color images being provided for each depth plane. Although the waveguides associated with each depth plane are shown adjacent to each other in this figure for ease of description, it should be understood that in a physical device, the waveguides may all be arranged as a stack with one waveguide per layer. In some other embodiments, multiple component colors may be output by the same waveguide, such that, for example, only a single waveguide may be provided for each depth plane.

With continued reference to fig. 8, in some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light may be used in addition to or in place of one or more of red, green, or blue (including magenta and cyan).

It should be understood that reference throughout this disclosure to light of a given color will be understood to include light perceived by a viewer as having one or more wavelengths within the range of light wavelengths of the given color. For example, red light may include one or more wavelengths of light in the range of about 620nm to 780nm, green light may include one or more wavelengths of light in the range of about 492nm to 577nm, and blue light may include one or more wavelengths in the range of about 435nm to 493 nm.

In some embodiments, light source 530 (fig. 6) may be configured to emit light at one or more wavelengths (e.g., infrared and/or ultraviolet wavelengths) outside of the visual perception range of the viewer. Further, the incoupling, outcoupling and other light redirecting structures of the waveguide of the display 250 may be configured to guide this light out of the display and emit towards the eye 210 of the user, e.g. for imaging and/or user stimulation applications.

Referring now to fig. 9A, in some embodiments, it may be desirable to redirect light impinging on the waveguide to couple the light into the waveguide. Incoupling optical elements may be used to redirect and couple light into their corresponding waveguides. Fig. 9A illustrates a cross-sectional side view of an example of a plurality or set of stacked waveguides 660, where each stacked waveguide includes an incoupling optical element. The waveguides may each be configured to output light of one or more different wavelengths, or one or more different wavelength ranges. It should be understood that the stack 660 may correspond to the stack 260 (fig. 6), and that the waveguides of the illustrated stack 660 may correspond to a portion of the plurality of

waveguides

270, 280, 290, 300, 310, except that light from one or more of the

image injection devices

360, 370, 380, 390, 400 is injected into the waveguides from a location that needs to be redirected to be coupled in.

Stacked waveguide set 660 is shown to include

waveguides

670, 680, and 690. Each waveguide includes associated incoupling optical elements (which may also be referred to as light input regions on the waveguide), such as incoupling optical element 700 disposed on a major surface (e.g., a top major surface) of waveguide 670, incoupling optical element 710 disposed on a major surface (e.g., a top major surface) of waveguide 680, and incoupling optical element 720 disposed on a major surface (e.g., a top major surface) of waveguide 690. In some embodiments, one or more of the incoupling

optical elements

700, 710, 720 may be disposed on the bottom major surface of the

respective waveguide

670, 680, 690 (particularly if one or more of the incoupling optical elements are reflective deflecting optical elements). As shown, the incoupling

optical elements

700, 710, 720 may be disposed on the top major surface of their

respective waveguides

670, 680, 690 (or the top of the next layer of waveguides), particularly if these incoupling optical elements are transmissive deflecting optical elements. In some embodiments, the incoupling

optical elements

700, 710, 720 may be disposed in the volume of the

respective waveguides

670, 680, 690. In some embodiments, the incoupling

optical elements

700, 710, 720 are wavelength selective, as discussed herein, such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths. Although shown on one side or corner of their

respective waveguides

670, 680, 690, it should be understood that in some embodiments, incoupling

optical elements

700, 710, 720 may be disposed in other regions of their

respective waveguides

670, 680, 690.

The incoupling

optical elements

700, 710, 720 may be laterally offset from one another as shown. In some embodiments, each incoupling optical element may be offset such that it receives light without passing through another incoupling optical element. For example, each incoupling

optical element

700, 710, 720 may be configured to receive light from a different

image injection device

360, 370, 380, 390, and 400, as shown in fig. 6, and may be separated (e.g., laterally spaced) from the other incoupling

optical elements

700, 710, 720 such that the incoupling optical element does not substantially receive light from the other ones of the incoupling

optical elements

700, 710, 720.

Each waveguide also includes associated light distribution elements, e.g., light distribution element 730 disposed on a major surface (e.g., top major surface) of waveguide 670, light distribution element 740 disposed on a major surface (e.g., top major surface) of waveguide 680, and light distribution element 750 disposed on a major surface (e.g., top major surface) of waveguide 690. In some other embodiments,

light distribution elements

730, 740, 750 may be arranged on a bottom main surface of associated

waveguides

670, 680, 690, respectively. In some other embodiments,

light distribution elements

730, 740, 750 may be provided on the top and bottom major surfaces of associated

waveguides

670, 680, 690, respectively; alternatively, the

light distributing elements

730, 740, 750 may be arranged on different ones of the top and bottom main surfaces in different associated

waveguides

670, 680, 690, respectively.

The

waveguides

670, 680, 690 may be spaced apart and separated by layers of, for example, gas, liquid, and/or solid material. For example, as shown, layer 760a may separate

waveguides

670 and 680; layer 760b may separate

waveguides

680 and 690. In some embodiments,

layers

760a and 760b are formed of a low index material (i.e., a material having a lower index of refraction than the material forming the immediately adjacent one of

waveguides

670, 680, 690). Preferably, the refractive index of the

material forming layers

760a, 760b differs from the refractive index of the

material forming waveguides

670, 680, 690 by 0.05 or more, or by 0.10 or less. Advantageously, the

low index layers

760a, 760b may act as cladding layers that promote Total Internal Reflection (TIR) of light through the

waveguides

670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the

layers

760a, 760b are formed of air. Although not shown, it is understood that the top and bottom of the illustrated waveguide set 660 can include immediately adjacent cladding layers.

Preferably, for ease of manufacturing and for other considerations, the

materials forming waveguides

670, 680, 690 are similar or the same, and the

materials forming layers

760a, 760b are similar or the same. In some embodiments, the

material forming waveguides

670, 680, 690 may be different between one or more waveguides, and/or the

material forming layers

760a, 760b may be different, while still maintaining the various refractive index relationships described above.

With continued reference to FIG. 9A,

light rays

770, 780, 790 are incident on waveguide set 660. It should be appreciated that the light rays 770, 780, 790 may be injected into the

waveguides

670, 680, 690 by one or more

image injection devices

360, 370, 380, 390, 400 (FIG. 6).

In some embodiments, the light rays 770, 780, 790 are intended for different waveguides (e.g., waveguides configured to output light having different amounts of wave front divergence and/or configured to output light having different characteristics, such as different wavelengths or colors). Thus, in some embodiments, the light 770, 780, 790 may have different properties, e.g., different wavelengths or different wavelength ranges corresponding to different colors. The incoupling

optical elements

700, 710, 720 each deflect incident light such that the light propagates through a respective one of the

waveguides

670, 680, 690 by TIR. In some embodiments, the incoupling

optical elements

700, 710, 720 each selectively deflect one or more particular wavelengths of light while transmitting other wavelengths to the underlying waveguide and associated incoupling optical element.

For example, the incoupling optical element 700 may be configured to deflect light 770 having a first wavelength or wavelength range while transmitting light 780 and light 790 having different, second and third wavelengths or wavelength ranges, respectively. The transmitted light 780 impinges on and is deflected by the incoupling optical element 710, which incoupling optical element 710 is configured to deflect light of a second wavelength or wavelength range. The light 790 is deflected by the incoupling optical elements 720, the incoupling optical elements 720 being configured to selectively deflect light of a third wavelength or wavelength range.

With continued reference to fig. 9A, the deflected

light rays

770, 780, 790 are deflected such that they propagate through the corresponding

waveguides

670, 680, 690; that is, the incoupling

optical elements

700, 710, 720 of each waveguide deflect light into the corresponding

waveguide

670, 680, 690 to couple the light into the corresponding waveguide. Light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the

respective waveguides

670, 680, 690 by TIR. Light rays 770, 780, 790 propagate through the

respective waveguide

670, 680, 690 by TIR until impinging on the corresponding

light distribution element

730, 740, 750 of the waveguide.

Referring now to fig. 9B, a perspective view of an example of the multiple stacked waveguides of fig. 9A is shown. As described above, the

incoupling light

770, 780, 790 is deflected by the incoupling

optical elements

700, 710, 720, respectively, and then propagates within the

waveguides

670, 680, 690, respectively, by TIR.

Light

770, 780, 790 then impinges on

light distribution element

730, 740, 750, respectively. The

light distributing elements

730, 740, 750 deflect the light rays 770, 780, 790 such that they propagate towards the outcoupling

optical elements

800, 810, 820, respectively.

In some embodiments, the

light distribution elements

730, 740, 750 are Orthogonal Pupil Expanders (OPE). In some embodiments, the OPE deflects or distributes light to the outcoupling

optical elements

800, 810, 820, and in some embodiments, may also increase the beam or spot size of the light as it propagates towards the outcoupling optical elements. In some embodiments, the

light distribution elements

730, 740, 750 may be omitted and the incoupling

optical elements

700, 710, 720 may be configured to deflect light directly into the outcoupling

optical elements

800, 810, 820. For example. Referring to fig. 9A, the

light distribution elements

730, 740, 750 may be replaced by outcoupling

optical elements

800, 810, 820, respectively. In some embodiments, the outcoupling

optical elements

800, 810, 820 are Exit Pupils (EP) or Exit Pupil Expanders (EPE) that direct light into the eye 210 (fig. 7) of the viewer. It should be understood that the OPE may be configured to increase the size of the eye movement range (eye box) in at least one axis, and the EPE may increase the size of the eye movement range in an axis that is orthogonal (e.g., orthogonal) to the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light illuminating the OPE to an EPE of the same waveguide while allowing the remainder of the light to continue propagating along the waveguide. When the OPE is illuminated again, another portion of the remaining light is redirected to the EPE, and the remainder of that portion continues to propagate further along the waveguide, and so on. Similarly, upon illuminating the EPE, a portion of the illumination light is directed out of the waveguide toward the user, and the remainder of the light continues to propagate through the waveguide until it again illuminates the EP, at which point another portion of the illumination light is directed out of the waveguide, and so on. Thus, each time a portion of the light is redirected by an OPE or EPE, a single beam of incoupled light can be "replicated" to form a field that clones the light beam, as shown in fig. 6. In some embodiments, the OPE and/or EPE may be configured to modify the size of the light beam.

Thus, referring to fig. 9A and 9B, in some embodiments, waveguide set 660 includes

waveguides

670, 680, 690 for each component color; coupling

optical elements

700, 710, 720; light distribution elements (e.g. OPE)730, 740, 750; and outcoupling optical elements (e.g., EP)800, 810, 820. The

waveguides

670, 680, 690 may be stacked with an air gap/cladding between each waveguide. The incoupling

optical elements

700, 710, 720 redirect or deflect incident light (where different incoupling optical elements receive light of different wavelengths) into their waveguides. The light then propagates at an angle, which results in TIR within the

respective waveguide

670, 680, 690. In the example shown, light 770 (e.g., blue light) is deflected by first incoupling optical element 700 in the manner previously described, and then continues to bounce along the waveguide, interacting with light distribution element (e.g., OPE)730 and outcoupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red, respectively) will propagate through waveguide 670, where light ray 780 impinges on incoupling optical element 710 and is deflected by incoupling optical element 710. Light ray 780 then bounces via TIR along waveguide 680, continues to propagate to its light distribution element (e.g., OPE)740, and then reaches out-coupling optical element (e.g., EP) 810. Finally, light 790 (e.g., red light) propagates through waveguide 690 and impinges on the incoupling optical elements 720 of waveguide 690. Incoupling optical element 720 deflects light 790 such that the light propagates by TIR to light distribution element (e.g., OPE)750 and then to outcoupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light 790 to a viewer, who also receives out-coupled light from the

other waveguides

670, 680.

Fig. 9C illustrates a top plan view of an example of the multiple stacked waveguides of fig. 9A and 9B. As shown, the

waveguides

670, 680, 690 and the associated

light distribution elements

730, 740, 750 and the associated outcoupling

optical elements

800, 810, 820 of each waveguide may be vertically aligned. However, as discussed herein, the incoupling

optical elements

700, 710, 720 are not vertically aligned; in contrast, the incoupling optical elements are preferably non-overlapping (e.g., laterally spaced apart as viewed in plan). As discussed further herein, this non-overlapping spatial arrangement facilitates one-to-one injection of light from different resources into different waveguides, thereby allowing a particular light source to be uniquely coupled to a particular waveguide. In some embodiments, arrangements comprising non-overlapping spatially separated incoupling optical elements may be referred to as shifted pupil systems, and the incoupling optical elements within these arrangements may correspond to sub-pupils.

Fig. 9D illustrates an example of a wearable display system 60 in which the various waveguides and related systems disclosed herein may be integrated. In some embodiments, the display system 60 is the system 250 of fig. 6, with fig. 6 schematically illustrating portions of the system 60 in more detail. For example, the waveguide assembly 260 of fig. 6 may be part of the display 70.

With continued reference to fig. 9D, the display system 60 includes a display 70, as well as various mechanical and electronic modules and systems that support the functionality of the display 70. The display 70 may be coupled to a frame 80 that is wearable by a display system user or viewer 90 and is configured to position the display 70 in front of the eyes of the user 90. In some embodiments, the display 70 may be considered glasses. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned near the ear canal of the user 90 (in some embodiments, another speaker (not shown) may be selectively positioned near the other ear canal of the user to provide stereo/shapeable sound control). The display system 60 may also include one or more microphones 110 or other devices that detect sound. In some embodiments, the microphone is configured to allow a user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.) and/or may allow audio communication with others (e.g., with other users of similar display systems). The microphone may be further configured as a peripheral sensor to collect audio data (e.g., sounds from the user and/or the environment). In some embodiments, the display system may also include peripheral sensors 120a, which may be separate from the frame 80 and attached to the body of the user 90 (e.g., on the head, torso, limbs, etc. of the user 90). In some embodiments, the peripheral sensor 120a may be configured to acquire data characterizing a physiological state of the user 90. For example, the sensor 120a may be an electrode.

With continued reference to fig. 9D, the display 70 is operatively coupled to a local data processing module 140 by a communication link 130 (such as by a wired lead or wireless connection), and the local data processing module 140 may be mounted in various configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration, in a strapped configuration). Similarly, sensor 120a may be operatively coupled to local processing and data module 140 via communication link 120b (e.g., via wired leads or a wireless connection). The local processing and data module 140 may include a hardware processor, as well as digital memory, such as non-volatile memory (e.g., flash memory or a hard drive), both of which may be used to facilitate data processing, caching, and storage. Alternatively, the local processor and data module 140 may include one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), dedicated processing hardware, and the like. These data include a) data captured by sensors (e.g., which may be operatively coupled to the frame 80 or otherwise attached to the user 90), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, gyroscopes, and/or other sensors disclosed herein; and/or b) data (including data relating to virtual content) obtained and/or processed using remote processing module 150 and/or remote data store 160, which may be transmitted to display 70 after the above-described processing or retrieval has been performed. The local processing and data module 140 may be operatively coupled to the remote processing module 150 and the remote data store 160 by communication links 170, 180 (such as via wired or wireless communication links) such that these

remote modules

150, 160 are operatively coupled to each other and may serve as resources for the local processing and data module 140. In some embodiments, the local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio, and/or a gyroscope. In some other embodiments, one or more of these sensors may be attached to the frame 80, or may be a separate structure that communicates with the local processing and data module 140 through a wired or wireless communication path.

With continued reference to fig. 9D, in some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and/or image information, e.g., including one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), dedicated processing hardware, and the like. In some embodiments, the remote data store 160 may include a digital data storage facility, which may be available through the internet or other network configuration in a "cloud" resource configuration. In some embodiments, the remote data store 160 may include one or more remote servers that provide information, such as information used to generate augmented reality content, to the local processing and data module 140 and/or the remote processing module 150. In some embodiments, storing all data and executing all data in the local processing and data module allows for fully autonomous use from the remote module. Alternatively, an external system (e.g., a system of one or more processors, one or more computers) including a CPU, GPU, etc. may perform at least part of the processing (e.g., generating image information, processing data) and provide information to

modules

140, 150, 160 and receive information from

modules

140, 150, 160, e.g., via a wireless or wired connection.

Enhanced eye pose determination techniques

The display systems described herein (e.g., display system 60, fig. 9D) can be used to present augmented or virtual reality content (referred to herein as virtual content). To render virtual content, the display system may monitor the eye pose of the user's eyes. As described herein, eye gestures may indicate an orientation of an eye, which may be used to identify various parameters, such as a particular axis of the eye (e.g., an optical axis and/or a visual axis). Described below are techniques that may be applied in some embodiments to increase the speed at which eye pose may be determined, while additionally reducing power requirements and enabling mechanical efficiency.

As described above, determining eye gestures may be used for various purposes to improve the viewing experience and functions associated with the presentation of virtual content. For example, the eye gestures may inform the display system how to render the virtual content. In this example, the display system may place the rendering cameras at respective centers of the user's eyes to provide a correct view of the virtual content for presentation to the user. In addition, the display system can use a fast eye pose determination to reduce accommodation-convergence mismatch. As described in fig. 4A through 4D, a vergence-accommodation mismatch may exist in a display system including a limited number of depth planes. Each of the depth planes may correspond to a particular depth range from the user. Each depth plane has an associated adjustment cue, and there may be available adjustment cues corresponding to a limited number of depth planes. Instead, the vergence cues may be updated via adjusting the dichotomous presentation of the virtual content. Thus, the accommodative cues may be the same throughout the depth range over which the vergence cues may be adjusted. The accommodation cues provided to the user may be selected based on the data plane that their eyes are approaching. By providing accommodative cues that do not correspond to the current convergence point (and plane) of the eyes, a determination of convergence that lags behind the actual pose of the eyes can undesirably introduce accommodation-convergence mismatch. By accurately determining the eye pose in real time, the display system can determine the depth plane that the eyes are approaching, allowing the display system to provide accommodative clues for that depth plane to maintain the accommodative-vergence mismatch at a low level.

Advantageously, using the techniques described herein, eye pose may be determined at substantially higher frequencies (e.g., 1kHz, 10kHz, etc.) than some prior art techniques. Further, this determination may be accomplished using components that require less power than some prior art techniques. For example, light (e.g., infrared or near-infrared light) may be directed from a light source (e.g., a diode, such as a VCSEL) onto the user's eye. In some embodiments, the light may be reflected by a movable reflector having a diffraction grating, wherein the diffraction grating provides a pattern of light. In some other embodiments, a light pattern may be formed at the light source (e.g., using a diffraction grating provided on an output surface of the light source). The movable reflector may be rotated about one or more axes such that the reflected light is scanned (e.g., swept) over the user's eye. In some embodiments, the light may be scanned across the horizontal axis of the user's eyes, the horizontal axis extending from the center of one eye to the center of the other eye. As mentioned above, in some embodiments, the movable reflector may be rotated about a single axis such that the reflected light is scanned (e.g., swept) over the user's eye in one dimension. For example, in at least some of these embodiments, the movable reflector may be configured and/or controlled such that the position of the reflected light pattern varies temporally in a single first dimension (e.g., in a horizontal axis of the user's eye), while the light pattern itself exhibits spatial variation in a second, different dimension (e.g., in a vertical axis of the user's eye). It should also be understood that in these embodiments, the movable reflector may not be configured to rotate about more than one axis, or the movable reflector may be configured to rotate about two or more axes, but controlled in some manner to rotate about only one of the axes. It should also be understood that in these embodiments, the light pattern itself may exhibit spatial variation in the first and second dimensions. Although the particular axis described herein is a horizontal axis, in some other embodiments, light may be scanned across a vertical axis of the user's eye.

The light reflected by the eye during this scan may be measured by a threshold number of light detectors (e.g., photodiodes) positioned with respect to the display system. For example, the light intensity of the reflected light may be measured. When scanning light over the eye, a corresponding light intensity pattern corresponding to the light intensity detected over a period of time may be determined. In some embodiments, the light intensity pattern may be formed based on the current generated by each light detector. As described herein, non-uniformities in the eye may result in light intensity patterns that are unique to different eye poses, and thus the eye pose may be determined based on analysis of these light intensity patterns. For example, the light intensity pattern may match the expected pattern for different poses.

Referring now to fig. 10A, a system and technique for determining the eye pose of a user is shown in plan view. In the illustrated example, the

eyes

1002A, 1002B of the user are represented. The user may be a user of a display system, such as display system 60 (fig. 9D), and may be viewing virtual content presented via the display system. Thus, the illustrated system may be understood as part of the display system 60. For example, the light source 1302, the movable reflector 1006, and the light detector 1014 may be attached to the frame 80 of the display system 60. However, for ease of illustration and discussion, the frame 80 and the remainder of the display system 60 are not shown. Additionally, while a system for detecting the orientation of the eye 1002A is shown, it should be understood that a similar system may be provided for the eye 1002B and the orientation of the eye 1002B may be determined as discussed herein for the eye 1002A.

To determine the eye pose of the eye 1002A, the eye illumination system 1003 may be configured to direct light 1010 onto the eye 1002A and scan the light 1010 over the eye 1002A. In some embodiments, the eye illumination system 1003 includes a light source 1302 that directs light 1004 onto a movable reflector 1006, the movable reflector 1006 reflecting the light onto the eye 1002A. In some embodiments, the movable reflector 1006 may be a micro-electromechanical system (MEMS) mirror. As mentioned above, in some embodiments, the movable reflector 1006 may be a one-dimensional MEMS scanning mirror. In some embodiments, the light source 1302 may be a diode that emits light. As an example, a Vertical Cavity Surface Emitting Laser (VCSEL) may be used to output light 1004. In some embodiments, other diodes, other lasers including other coherent light sources, and the like may be used.

In some embodiments, the reflective surface of the movable reflector 1006 may comprise a diffraction grating 1008. Light 1004 may be reflected by diffraction grating 1008 on movable reflector 1006 such that light pattern 1010 is formed. In some embodiments, light 1004 may be polychromatic light (e.g., infrared and/or near-infrared light). Light of different wavelengths (or colors) that form polychromatic light can be diffracted in different directions, thereby generating pattern 1010. In the example of fig. 10A, the light pattern 1010 includes two

portions

1010A, 1010B that propagate in different directions away from the movable reflector 1006.

In some embodiments, the incident light 1004 may be monochromatic light (e.g., infrared or near-infrared light). The two

portions

1010A, 1010B may be formed using a suitably configured diffraction grating (e.g., a diffraction grating may comprise multiple portions having different orientations, sizes, geometries, etc. to achieve diffraction in a desired direction).

In the elevational view, the two

portions

1010A, 1010B are shown as being projected from the movable reflector 1006 toward the eye 1002A. However, it should be understood that the

portions

1010A, 1010B may be configured such that they form a row of light or spots that span the eye 1002A in a vertical direction when incident on the eye 1002A. For example, the

portions

1010A, 1010B may extend from a lower portion of the eye 1002A to an upper portion of the eye 1002A. A different perspective view of the light pattern is shown in fig. 10B and will be described in more detail below.

With continued reference to fig. 10A, the movable reflector 1006 may be controlled by the display system to move and cause the light pattern 1010 to be scanned across the eye 1002A along a horizontal axis. For example, and with respect to fig. 10A, the light pattern 1010 may scan from a left portion of the eye 1002A to a right portion of the eye 1002A. In some embodiments, the light pattern 1010 may be scanned across the sclera and over the iris and pupil of the eye 1002A from one side of the eye to the other side of the eye. In some embodiments, the extent of the scan may be more limited. For example, in some embodiments, the display system may be configured to scan the light pattern 1010 over the entire iris of the eye 1002A, but less than the entire extent of the eye along the axis of scan progression. The movable reflector 1006 may be adjusted such that it rotates about one or more axes. As mentioned above, in some embodiments, the movable reflector 1006 may be adjusted such that it rotates about a single axis. It should be appreciated that this rotation may adjust the angle at which light 1004 is incident on the diffraction grating 1008, thereby changing the direction in which light is reflected to the eye 1002A and changing the final position at which light is incident on the eye 1002A. Thus, the light pattern 1010 may be scanned across the eye 1002A by movement of the reflector 1006.

As shown, the light detector 1014 may receive reflected light 1012. It should be appreciated that reflected light 1012 is the portion of light 1010A, 1010B that is incident on eye 1002A and reflected from eye 1002A to photodetector 1014. As movable reflector 1006 scans light pattern 1010 across the horizontal axis of eye 1002A, reflected light 1012 may be received by light reflector 1014. It should be understood that different physiological characteristics of eye 1002A may reflect light differently. For example, the cornea 1016 may protrude from the rest of the eye to reflect light in a different direction than other parts of the eye. In addition, different parts of the eye may have different reflectivities. For example, the sclera (the "white of the eye") may be understood to reflect more light than the iris, which may reflect more light than the pupil. As another example, different portions of the eye may be associated with diffuse or specular reflection. As an example, the sclera may cause diffuse reflection such that the resulting light intensity pattern, such as shown in fig. 10B, may include light intensity peaks that are more gradually increasing or decreasing in intensity, and reach a lower maximum intensity value than specular reflection. In contrast, specular reflection may be associated with "bright spots" and result in a sharp increase or decrease in intensity. Differentiating between diffuse and specular light reflections may enable differentiation between reflections associated with "bright spots" and portions of the eye, such as the sclera, iris, and pupil (providing diffuse reflections), which may be used to identify the eyeball/corneal curvature. Thus, the reflected light 1012 may vary in intensity during the scan, where the variation in intensity is caused by the eye characteristics upon which the light 1010A, 1010B is incident. Depending on the orientation of the eye, these changes may be expected to occur at different points (e.g., at different times) during the scan. The pattern of light intensity provided by the reflected light 1012 may thus represent a signature representative of the particular eye pose of the eye 1002A.

The photo-detector 1014 may convert the reflected light 1012 into an electrical current. The display system may store information identifying the current (or values derived from the current) as plotted against time or the position of the movable reflector 1006 or incident light 1010A, 1010B, or may simply have certain current values (or values derived from the current) associated with particular times and/or positions (or values derived from particular times and/or positions). Such a map or sequence of values may be referred to herein as a light intensity pattern. Fig. 10B shows an example light intensity pattern. Preferably, multiple light intensity patterns derived from light detectors at different locations are utilized to increase the accuracy of the final determination of eye pose (or other eye parameter, as discussed herein).

Referring again to fig. 10A, the illustrated example may include a plurality of light detectors 1014, which are schematically represented as a single block. However, it should be understood that light detector 1014 may be located at a different location in the display system. For example, the system may include a plurality of light detectors arranged in a linear array or at corners of various shapes (e.g., four light detectors positioned in a rectangular configuration with respect to the user's eye 1002A).

Preferably, the photodetectors are positioned in front of the eye 1002A so that they receive the reflected light 1012 during the scan. Fig. 11 shows an example of such a configuration. It should be understood that the light intensity pattern of fig. 10B is an example of a single light intensity pattern detected using a single light detector 1014. Each of the illustrated photodetectors 1104A-1104D may have different light intensity patterns due to different positioning, such that they receive light 1012 reflected in different directions.

Thus, each of the photodetectors may generate a light intensity pattern associated with scanning the light pattern 1010 over the eye 1002A, e.g., on the horizontal axis. As discussed herein, the light intensity pattern may be used by the display system to determine an eye pose for the eye 1002A. In some embodiments, after scanning in the horizontal axis, the light pattern 1010 may be scanned in the opposite direction. Thus, two scans may optionally be performed. These two scans may be used by the display system to determine the eye pose for the eye 1002A. In some embodiments, the movable reflector 1006 may generally cause the light pattern 1010 to be scanned in the opposite direction only when a subsequent eye pose is determined. Thus, the eye pose for the eye 1002A may be determined based on a single scan of the eye 1002A in a direction in the horizontal axis (e.g., left to right). Subsequent eye poses for the eye 1002A can then be determined based on scanning in opposite directions along the horizontal axis (e.g., from right to left). Thus, the movable reflector may not need to be reset to the same position for each scan, which results in scanning from the same initial position along the same horizontal direction. In this way, allowing the scanning to occur in the opposite direction may increase the speed at which eye tracking occurs. For example, the eye tracking speed may be doubled compared to requiring scanning in the same direction determined for each eye pose.

The display system may then analyze the light intensity pattern to determine an eye pose for the eye 1002A. For example, the positions of light detector 1014 and movable reflector 1006 may be known to the display system (e.g., via initial calibration). As another example, light detector 1014 may be located the same as the light detector used to generate information that may be used to determine eye pose. As an example of determining eye pose, each light intensity pattern may represent a characteristic pattern associated with the orientation of the eye 1002A. Therefore, the aggregation of these light intensity patterns can be used to determine a particular eye pose with high accuracy.

In some embodiments, a machine learning model may be generated that outputs information identifying eye gestures based on a threshold number of inputs of light intensity patterns. In this example, the machine learning model may have been trained based on the same or similar placement of the light detector and the movable reflector. For example, light intensity patterns corresponding to known eye gestures may have been generated using similar placement of light detectors and movable reflectors within the display system. In this example, the machine learning model may then have been trained. As another example, the light intensity pattern along with information identifying the placement of the light detector and the movable reflector (e.g., relative to the eye) may have been used as training information. In some embodiments, the training information may additionally indicate the position of the light source and/or the movable reflector.

In some embodiments, the display system may store information identifying the light intensity pattern for each of a plurality of eye gestures. Thus, the display system may determine a similarity metric between the measured light intensity pattern from light detector 1014 and the stored light intensity pattern. These similarity measures may include measuring similarity in terms of peaks, valleys, slopes, curve fitting techniques, and the like.

As an example, the light intensity pattern measured by a particular photodetector may include different peaks and valleys. These peaks and valleys may correspond to respective times at which the two

portions

1010A, 1010B of the light pattern 1010 reflect from the eye 1002A or the position of the movable reflector 1006. Thus, there may be an "alpha" 1010A peak at a particular time or movable reflector 1006 position. In addition, there may be a "β" 1010B peak at a later time or at the movable reflector 1006 position. The display system may determine the eye pose based on these respective peaks. For example, the display system may identify the time of each peak or the movable reflector position. The display system may use these identified times or MEMS positions for each of the photodiodes to match a reference light intensity pattern corresponding to a known eye pose.

Referring again to FIG. 10B, an example of a light intensity pattern 1022 associated with scanning the user's eye 1002A with a light pattern 1010 is shown. As depicted in fig. 10A, movable reflector 1006 may form a light pattern 1010 and direct the pattern onto the user's eye 1002A using diffraction grating 1008. Photodetector 1014 may then receive reflected light 1012 from eye 1002A. As described above, there may be a threshold number of photodetectors 1014 (e.g., 2, 3, 4, 6, or more photodetectors) in different locations with respect to eye 1002A. Thus, each light detector may convert the respective received light into a current or signal (e.g., a digital or analog signal). This current or signal may be measured by the display system and stored as a light intensity pattern 1022.

In the illustrated example, a light pattern 1010 (e.g., a "V" pattern) is provided to the user's eye 1002A at a first location. As shown, the light pattern 1010 may extend along a vertical axis of the user's eye 1002A. Further, the light pattern 1010 may include an "alpha" portion 1010A angled opposite to the angle of the "beta" portion 1010B. Thus, the

portions

1010A, 1010B may be projected on different portions of the user's eye 1002A at any given time during the scan. These two

portions

1010A, 1010B may be used to inform the eye 1002A of the eye's eye pose, as described below. In some embodiments, to distinguish the different reflected light signals provided by each

portion

1010A, 1010B, the portions may each be formed from different colors of light (from incident polychromatic light, as discussed herein). In some other embodiments, the

different portions

1010A, 1010B may be formed from the same color of light and may be generated at different times by illuminating different portions of the movable reflector 1006, which may have different diffraction gratings in these different portions. Generating the

portions

1010A, 1010B at different times allows the different signals provided by the

portions

1010A, 1010B to be distinguished in time. In some other embodiments, the

portions

1010A, 1010B may be formed from the same color of light and may be scanned simultaneously on the eye.

With continued reference to FIG. 10b, at an initial time (e.g., time t0), the movable reflector 1006 may be at the end of its range of rotation. For example, the movable reflector 1006 may cause the light pattern 1010 to be at a leftmost or rightmost position on the eye 1002A. In the illustrated example, the initial time corresponds to the light pattern 1010 being projected onto the leftmost portion of the eye 1002A. Light reflected from the eye 1002A may be measured at this location. An example of such an initial measurement 1024A is reflected in the light intensity pattern 1022 in fig. 10B. The light pattern 1010 may be continuously or discretely moved over the eye 1002A by the movable reflector 1006.

At a subsequent time 1020C, and thus at a different movable reflector 1006 position, the light pattern 1010 may move as shown in fig. 10B. At this position of the movable reflector 1006, the "alpha" portion 1010A of the light pattern 1010 has reached the end of the pupil of the eye 1002A (e.g., the rightmost portion). As shown in light intensity pattern 1022, the corresponding light detector associated with pattern 1022 is positioned such that at a subsequent time 1020C, a peak 1024C (caused by a bright spot of the eye) is represented in light intensity pattern 1022. In contrast, the valley 1024B is included in the light intensity pattern 1022 at an earlier time. For example, the valley 1024B may represent the movable reflector 1006 such that the "a" portion 1010A reaches the opposite end of the pupil of the eye 1002A (e.g., the leftmost portion). For this movable reflector 1006 position corresponding to valley 1024B, the corresponding light detector associated with pattern 1022 may have limited visibility of reflected light and/or light reflected to the light detector may be reflected from a portion of the eye having low reflectivity. Thus, the valley 1024B may be represented in the light intensity pattern 1022.

The light intensity pattern 1022 shows another peak 1024D at another time (e.g., at a further adjustment of the movable reflector 1006). The peak 1024D may correspond to a "bright spot" (e.g., specular reflection) with a bright spot at the same location that results in the peak 1024C. This example peak 1024D may be generated based on the trailing "β" portion 1010B of the light pattern 1010. For example, the corresponding light detector associated with pattern 1022 may have substantially maximum visibility of light reflected from "β" portion 1010B at the other time. As a non-limiting example, this peak 1024D may have been generated when the "β" portion 1010B passed the end of the pupil of the eye 1002A. This end may be the same end that provides the bright spot that causes the peak of the "alpha" portion 1010A. In this example, peak 1024D may correspond to "β" portion 1010B through the end and peak 1024C may correspond to "α" portion 1010A through the same end.

With continued reference to fig. 10b, the processing device 1026 may receive the light intensity pattern 1022 and use the pattern 1022 to determine an eye pose for the eye 1002A. In some embodiments, processing device 1026 may represent or be included in local data and processing module 140 described above. The processing device 1026 may optionally obtain information identifying a direction associated with the scanning of the light pattern 1010. For example, the generated light intensity pattern 1022 may be based on whether the light pattern 1010 is moving in a particular direction along a horizontal axis. As described above, with respect to fig. 10A, for a first eye pose, the MEMS mirror 1006 may be adjusted such that the light pattern 1010 moves along a first direction. For a second eye pose, which is subsequently determined, the MEMS mirror 1006 may then be rotated in the opposite direction to cause the light pattern to move along the opposite scan direction. In this manner, the MEMS mirror 1006 may increase the eye tracking speed compared to requiring scanning in the same direction.

Processing device 1026 may obtain light intensity patterns from a plurality of light detectors. For example, where there are four photodetectors, there may be at least four light intensity patterns associated with the same scan of eye 1002A. Each of these light intensity patterns may include a unique pattern corresponding to the amount of reflected light incident on the associated photodetector. For example, peaks 1024C and valleys 1024B corresponding to "a" portions 1010A may be located at different times or at different light intensity patterns at the movable reflector locations. Thus, the processing device 1026 may use these light intensity patterns to identify the exact eye pose of the eye 1002A.

Due to the high speed at which the MEMS mirror 1006 can be adjusted, and the limited information included in the light intensity pattern 1022, the processing device 1026 can quickly determine the eye pose. In embodiments using machine learning models, the processing device 1026 may optionally compute a forward pass of the neural network. The neural network may optionally include one or more dense (e.g., fully connected) layers. For example, values corresponding to the current and associated time or movable reflector position may be provided to a neural network. The neural network may optionally include one or more convolutional layers, which exploit the time-series nature of the light intensity pattern. These neural networks may have been previously trained. For example, the training data may include known eye poses and corresponding light intensity patterns. In this example, the positions of the movable reflector 1006 and the photodiode may optionally be the same as or similar to the positions used to generate the light intensity pattern 1022. Other machine learning models may be used and fall within the scope of the present disclosure herein. For example, a support vector machine may be used.

In some embodiments, the processing device 1026 may access stored information identifying known light intensity patterns and associated eye gestures. Processing device 1026 may then correlate the measured light intensity pattern (e.g., light intensity pattern 1022) with the stored information. The peaks (e.g., peak 1024C) and valleys (e.g., valley 1024B) in the measured light intensity pattern may be correlated with the peaks and valleys in the stored information. As described above, the movable reflector 1006 may optionally scan over the eye 1002A along a first direction or an opposite second direction. Thus, the processing device 1026 may optionally invert (e.g., reflect) or otherwise apply a linear transformation to the stored or measured light intensity pattern depending on the direction of the scan.

For example, processing device 1026 may access a stored light intensity pattern for a particular light detector of light detectors 1014. In this example, the processing device 1026 may identify a particular stored light intensity pattern that is closest to the measured light intensity pattern for the particular light detector. Although such specific stored light intensity patterns may be identified based on a number of different metrics, in some embodiments, processing device 1026 may identify specific light intensity patterns having similarly located peaks and valleys. With respect to the light intensity pattern 1022, the processing device 1026 can identify a stored light intensity pattern having peaks 1024C-1024D and valleys 1024B at similar times and/or similar movable reflector 1006 locations.

Using these threshold numbers of light intensity patterns 1022, the processing device 1026 may thus determine the eye pose of the eye 1002A with high accuracy. As will be described further below, in some embodiments, the processing device 1026 may use the light intensity pattern to determine an interface between physiological features of the eye 1002A. For example, the iris-to-pupil interface may be determined by processing device 1026.

Referring again to FIG. 11, an example of the positions of the light detectors 1104A-1104D within the display system 1102 for determining eye pose is shown, as discussed above. The photodetectors 1104A-1104D may be, for example, photodiodes, phototransistors, photoresistors, or combinations thereof. The example representation is illustrated from a perspective of a user facing the display system 1102. For ease of reference, the user's eye 1002A is shown in perspective. A plurality of light detectors 1104A-1104D are positioned with respect to the user's eye 1002A. As mentioned herein, in some embodiments, the light detectors 1104A-1104D may be attached to a frame of the display system 1102. For example, each of the light detectors 1104A-1104D may be attached to a different portion of the frame surrounding the eyepiece. In other embodiments, the light detectors 1104A-1104D may be attached to and/or embedded in a layer of material located adjacent to the eyepiece (e.g., a protective cover for the eyepiece) or the eyepiece itself. The light may be directed and scanned over the eye 1002A as described above. For example, the light pattern may be created using a diffraction grating located on or otherwise adjustable by a movable reflector (e.g., a MEMS mirror).

Because of the different positions of light detectors 1104A and 110D, each light detector will receive varying irradiance as the light pattern is scanned across the eye. In this way, each light detector may generate a different current pattern. In some embodiments, the light detector may generate a digital or analog signal associated with the received irradiance. For a given eye pose, each light detector will create a light intensity pattern that may represent a signature associated with the eye pose. As described above, with respect to fig. 10A-10B, the measured light intensity patterns can therefore be used to quickly recognize eye gestures.

The illustration includes four light detectors 1104A-1104D positioned equidistantly apart in a rectangular pattern. However, it should be understood that different locations may be used. For example, the photodetectors 1104A-1104B may be positioned closer together. As described above, the display system 1102 may determine the eye pose based on a machine learning model or stored information identifying known light intensity patterns. This information may be generated based on a similarly configured display system 1102. Thus, other locations of the photo-detectors 1104A-1104D may be used. For example, any location where the light detector is capable of receiving the reflected light may be used. In addition, the number of light detectors may be a total number other than four, as discussed herein. The machine learning model or stored information may be generated according to a particular configuration of the light detector. If the display system 1102 also uses this same configuration, the display system 1102 may recognize eye gestures based on machine learning models or stored information.

Referring now to fig. 12, an example of a flow diagram of a process 1200 for determining an eye pose of a user's eye is shown. For convenience, process 1200 will be described as being performed by a display system (e.g., display system 60, fig. 9D) having one or more processors.

At block 1202, the display system causes a light pattern to be projected via a movable reflector (e.g., a MEMS mirror). As shown in fig. 10A, the display system may output light using a light source (e.g., VCSEL). The output light may be provided to a movable reflector, which may optionally have a diffraction grating on its reflective surface. The diffraction grating may cause a light pattern to be provided on the user's eye. In some other embodiments, the movable reflector is a specular reflector and the pattern may be formed upstream of the movable reflector, e.g., at the output of the light source.

At block 1204, the display system adjusts the movable reflector to scan the light pattern over the user's eye. The movable reflector may be in a first position such that the light pattern is projected at a first portion of the user's eye. For example, the light pattern may be projected at the leftmost portion of the user's eye. The display system may then cause the movable reflector to be adjusted such that the light pattern is projected along the axis. As described above, the axis may be a horizontal axis. The light pattern may additionally extend over a vertical portion of the user's eye. Further, the light pattern may optionally have a "V" shape formed by substantially continuous light rays, as shown in fig. 10B.

In some embodiments, the light pattern may have a shape that is different from a "V" shape formed by substantially continuous light rays. For example, and as shown in fig. 15 and discussed further with respect to fig. 15, the light pattern may include spots or dots that form two portions (e.g., "a" portion and "β" portion as described above) instead of lines. As another example, and as shown in fig. 16 and discussed further with respect to fig. 16, the light pattern may encode an optical function. The optical function may increase light cone diffusion or decrease spot size to better distinguish the signals received by the light detector.

Alternatively, the display system may initially scan the user's eye for interpupillary distance (IPD). The light projector can then be modulated so that it only illuminates the IPD area during scanning. For example, only the IPD area may be used, as well as all of the margin required to project light onto the user's eye in any eye orientation. In this way, display system power may be reduced since some users may have smaller IPD area than others. Thus, the display system may conform the scan to each user.

With continued reference to FIG. 12, at block 1206, the display system obtains a light intensity pattern from a threshold number of light detectors (e.g., photodiodes). Since the movable reflector causes the light pattern to be scanned over the user's eye, the light detector may generate a corresponding current or detector signal. The current or detector signal may be represented as a light intensity pattern as described above.

At block 1208, the display system determines an eye pose based on the obtained light intensity pattern. As depicted in fig. 10B, the display system may use machine learning techniques to assign eye gestures to light intensity patterns. The display system may also determine a similarity measure between the obtained light intensity pattern and the known light intensity pattern. In some embodiments, the known light intensity patterns may be stored as table data, where the current is plotted against time or movable reflector position. In some embodiments, the known light intensity pattern may be stored as information generated from analyzing the known light intensity pattern. For example, peaks, valleys, slopes, etc. for the light pattern may be stored. In this example, and with respect to the "V" pattern, the peaks and valleys for the "α" and "β" portions may be stored along with the corresponding positions of the movable mirrors.

In some embodiments, and as will be further described in fig. 18A-18D, the display system may determine an interface between different physiological portions of the eye. For example, the light intensity pattern may include information indicative of such an interface. When a light pattern is scanned across an interface, there may be a corresponding change in the current or signal generated by one or more photodetectors. An example interface may include an interface between an iris and a pupil. The display system may identify this example interface based on differences in light absorption and reflectance between the iris and the pupil. For example, there may be greater light absorption and lower reflectivity in the pupil. In this example, the resulting light intensity pattern may reflect a drop in current or signal as the light pattern crosses from the iris to the pupil.

With respect to the iris/pupil interface, the display system may determine the size and/or location of the pupil. For example, the display system may determine the leftmost interface and the rightmost interface as represented in one or more light intensity patterns. The display system may then determine the size of the pupil based on the difference between the leftmost interface and the rightmost interface. The determination may be based on a movable reflector. For example, the display system may identify a distance along a horizontal axis from a leftmost interface to a rightmost interface based on a rotational speed of the movable reflector.

Example eye gestures

Fig. 13A to 13C show scanning of a light pattern over a first eye pose of an eye. Similarly, fig. 14A to 14C show the patterns scanned over the second eye pose of the eye. As discussed herein, these different eye poses will cause different light intensity patterns to be generated.

Fig. 13A-13C illustrate examples of a light pattern 1010 being projected onto a user's eye 1002A and scanned over the user's eye 1002A. In the example shown, a light source 1302 (e.g., VCSEL diode) projects light 1004 onto a movable reflector 1006, which movable reflector 1006 may optionally have a diffraction grating 1008. The resulting light pattern 1010 then propagates to the eye 1002A. As shown, a portion of the light pattern is then reflected from the eye 1002A as reflected light 1012. As described above, a photodetector (not shown) may be positioned with respect to the eye 1002A to receive the reflected light 1012.

Fig. 13A-13C thus illustrate scanning of the light pattern 1010 over the user's eye 1002A by adjusting the position of the movable reflector 1006 (e.g., by rotating the movable reflector 1006). It should be appreciated that the reflected light 1012 is reflected in different directions depending on the position of the movable reflector 1006. Thus, the photodetectors positioned with respect to eye 1002A may each generate a unique light intensity pattern. In this way, the display system can determine a specific eye posture corresponding to the eye posture shown in fig. 13A to 13C.

For example, fig. 13A shows the movable reflector 1006 in an initial position. At this initial position, the light pattern 1010 is directed onto a corresponding initial position of the eye 1002A. For this eye pose, the initial position corresponds to the left portion of the eye 1002A. The reflected light 1012 for this initial position is shown as being reflected toward the left side of the eye 1002A. The reflected light 1012 may be received by a photodetector and may correspondingly generate a current or signal.

In fig. 13B, the movable reflector 1006 has scanned the light pattern 1010 to a substantially central portion of the eye 1002A. In this example, the reflected light 1012 is being reflected toward the left and right sides of the eye 1002A. In some embodiments, the light detector may be positioned with respect to the eye 1002A (e.g., as shown in fig. 11). Thus, the photodetector located on the right side of the eye 1002A may receive additional light as compared to the light received in fig. 13A.

In fig. 13C, movable reflector 1006 has scanned light pattern 1010 to the right portion of eye 1002A. The right portion may correspond to the final position of the movable reflector 1006. For this final position, the reflected light 1012 is being reflected toward the right side of the eye 1002A. Accordingly, the movable reflector 1006 may scan the light pattern 1010 via adjustment from the initial position to the final position. In some embodiments, movable reflector 1006 may scan light pattern 1010 by rotating about one or more axes. As mentioned herein, in some embodiments, the movable reflector 1006 may scan the light pattern 1010 by rotating about a single axis. The amount of rotation may be based on physiological characteristics of the eye 1002A. For example, the physiological characteristic may include the sclera, cornea, pupil, etc., and the amount of rotation may be based on the corresponding size of the characteristic (e.g., length along a horizontal axis).

Subsequently, the display system may obtain a light intensity pattern reflecting the scanning of the light pattern 1010 from the initial position to the final position. As described herein, these light intensity patterns may be used to determine the eye pose shown in fig. 13A-13C.

Fig. 14A-14C illustrate another example of a light pattern 1010 being projected onto a user's eye 1002A and scanned on the user's eye 1002A, where the user's eye 1002 is in a different pose than the eyes shown in fig. 13A-13C. In the eye pose shown, the center of the eye 1002A is angled toward the right side of the figure.

In fig. 14A, the movable reflector 1006 is in an initial position. For this eye pose, a light pattern 1010 is provided to the left part of the eye. Since the eye pose shown is angled to the right as compared to the eye pose shown in fig. 13A-13C, the light pattern 1010 is provided to a different portion of the eye 1002A. Therefore, the reflected light 1012 is reflected differently from the reflected light shown in fig. 13A. The photodetectors positioned with respect to eye 1002A will thus generate different measures of current or signal.

In fig. 14B, the movable reflector 1006 has been adjusted such that the light pattern 1010 is further scanned across the eye 1002A along the horizontal axis. For this example eye pose, the light pattern 1010 is closer to the cornea of the eye 1002A. In contrast, the eye posture shown in fig. 13B represents that the eye 1002A is looking straight ahead. Thus, in fig. 13B, a light pattern 1010 is provided to a substantially central portion of the eye 1002A. In fig. 14C, the movable reflector 1006 has been adjusted to a final position. As shown, the light pattern 1010 has been substantially scanned over the cornea of the eye 1002A.

It should be appreciated that the light pattern 1010 has been scanned across different portions of the eye 1002A as compared to the eye shown in fig. 13A-13C. Therefore, the generated light intensity pattern will be unique as compared to the light intensity pattern generated by the scanning shown in fig. 13A to 13C. For some eye poses, the light pattern 1010 may be scanned over a substantially similar portion of the eye 1002A. For example, the light pattern 1010 may be scanned over similar portions of the forward looking eye and the downward looking eye. However, these eyes will be in different orientations, such that the physiological characteristics of the eyes will be in different orientations. For example, the cornea would be in a different orientation. Thus, reflected light 1012 will result in a unique light intensity pattern being generated. In this way, the display system may determine the eye pose based on these light intensity patterns.

Example light Pattern

Fig. 15 shows an example of a light pattern 1502 for scanning across on a user's eye. In this example, the light pattern 1502 created by the diffraction grating 1008 consists of spots or spots of light, rather than continuous rays of light. Thus, the light pattern 1502 may represent a pattern displacement that may be projected onto a user's eye during adjustment of a movable reflector (e.g., reflector 1006, fig. 10). In some embodiments, multiple diffraction gratings with different pitches may be etched to superimpose different diffraction patterns and thus create "lines" or dots that are close enough to appear as lines.

Fig. 16 shows another example of a light pattern 1602 for scanning over a user's eye. In some embodiments, the light pattern 1602 may include multiple rows of light 1604, 1606, 1608, which may be formed by different light lines or spots forming separate rows. As shown, the

rows

1604, 1606, 1608 can each define an angle less than 90 ° relative to the horizontal axis of the eye. To distinguish the reflected light signals provided by each of the

rows

1604, 1606, 1608, the light forming the rows of light can have different characteristics. For example, the light forming different ones of the

rows

1604, 1606, 1608 may have different colors or wavelengths, different polarizations (e.g., in the case of using polarization sensitive photo detectors), and so forth.

In some other embodiments, the light forming the

different rows

1604, 1606, 1608 may have different associated optical functions. As an example, the optical function may increase light cone diffusion (e.g., may provide a diverging beam). As another example, the optical function may reduce the spot size (e.g., provide a converging light beam), which may be advantageous for providing a high signal-to-noise ratio for the light detector. As yet another example, the rows may be formed from collimated light beams. In some embodiments, the desired level of convergence, divergence, or collimation may be provided by a holographic material (e.g., a surface or volume HOE) on a movable reflector that provides lens work (e.g., collimating, focusing, or diverging lens function) energy and diffraction to form the desired line pattern.

Two light sources

In some embodiments, two light sources (e.g., two VCSELs) may be used to determine the eye pose of the eye. Each light source may output light having different characteristics (e.g., different wavelengths or colors). With respect to the example of a "V" pattern, a first light source may be used to form a first portion (e.g., "a" portion 1010A) and a second light source may be used to form a second portion (e.g., "β" portion 1010B) of the light pattern. The use of two light sources may provide example advantages, such as reduced cross talk. For example, there may be no crosstalk between the "α" portion and the "β" portion. As another example, the scan time may be reduced. For example, a movable reflector (e.g., MEMS mirror 1006) may be required to perform fewer adjustments (e.g., rotation about one or more axes) to scan a user's eye.

Fig. 17A shows an example of a light pattern 1010 projected onto a user's eye using two light sources. In the leftmost portion of the illustration, the light pattern has an "alpha" portion 1010A of the light pattern 1010 formed by the first light source. For example, "beta" portion 1010B is not projected by the second light source. The multiple photodetectors may thus measure the reflected light as the "alpha" portion 1010A is projected. Subsequently, and with respect to the rightmost portion of the illustration, a "β" portion 1010B is projected onto the user's eye. In contrast, "a" portion 1010A is not directed onto the user's eye. Thus, when the "β" portion 1010B is directed onto the user's eye, the plurality of photodetectors may measure the reflected light. In this way, cross talk between these parts may be reduced or eliminated.

The display system may optionally generate two light intensity patterns for each photodetector, one for the "alpha" portion 1010A and one for the "beta" portion 1010B. As an example, the display system may store information identifying the time and/or MEMS mirror position at which "alpha" portion 1010A or "beta" portion 1010B is being projected. In some embodiments, the display system may generate a single light intensity pattern representative of both the "α" portion 1010A and the "β" portion 1010B, and may optionally scan both

portions

1010A, 1010B on the eye simultaneously.

FIG. 17B illustrates an example block diagram that is shown using two

light sources

1702A, 1702B. In the illustrated example, light is provided to the movable reflector 1006, and the movable reflector 1006 may optionally have a diffraction grating 1008, as discussed herein. In some embodiments, the movable reflector 1006 may include a holographic element for generating the desired light rays (e.g., the hologram may be a multiplexed hologram including one hologram that is selective for one wavelength of light of line "α" and another hologram that is selective for the wavelength of light of line "β"). In some other embodiments, movable reflector 1006 is a specular reflector, and forms a desired pattern at

light sources

1702A, 1702B. Combiner 1704 may be used to direct light from

light sources

1702A, 1702B to movable reflector 1006.

Another embodiment may include having two regions on the movable reflector 1006. For each of the regions, there may be a particular diffraction grating, wherein each of the light sources is configured to illuminate one of the regions. Thus, different lines (e.g., "alpha" and "beta" portions) may be created. Fig. 17C shows an example of such a diffraction grating having a plurality of diffraction regions. Each diffractive zone may comprise a differently configured diffraction grating. For example, the diffraction gratings of the diffractive zones may have different physical parameters, including different periodicities and/or different sizes (e.g., heights and/or widths) of the individual structures (e.g., laterally extending lines of material) that form the gratings.

Determining the size and/or location of a physiological feature

It should be understood that the light intensity pattern may contain information that may be used to determine eye parameters other than pose. For example, the display system may be configured to determine the size and/or location of a physiological feature of the user's eye using the light intensity patterns described herein. Example physiological characteristics may include the sclera, the iris, the pupil, the interface between the sclera and the iris, the interface between the iris and the pupil, and so forth. The display system may determine size and/or position as an alternative or in addition to determining eye pose.

With respect to the interface between the iris and the pupil, the display system may determine its position based on changes in the current or signal as represented in one or more light intensity patterns. For example, the display system may identify a peak or a trough (e.g., a change in the derivative is greater than a threshold). With respect to the pupil, the display system may determine its size based on identifying the boundary of the pupil. As an example, the boundary may correspond to the leftmost interface and the rightmost interface between the pupil and the iris.

Fig. 18A shows an example flow diagram of a process 1800 for determining physiological information associated with an eye of a user. For convenience, process 1800 will be described as being performed by a display system (e.g., display system 60) of one or more processors.

At block 1802, and as described above with respect to block 1204 of fig. 12, the display system adjusts the movable reflector to scan the light pattern over the eye. At block 1804, the display system obtains a light intensity pattern from a threshold number of light detectors, similar to block 1206 of fig. 12. As described herein, the display system may obtain a light intensity pattern that represents a measure of the current or signal generated by the respective light detector. Each light intensity pattern may map a current or signal to a time and/or position of the movable reflector.

At block 1806, the display system determines a size and/or a location associated with a physiological characteristic of the eye. As discussed herein, in some embodiments, machine learning and/or pattern matching may be utilized to determine the location of physiological features or boundaries, and the size may be calculated from these determined locations.

18B-18D, FIGS. 18B-18D illustrate examples of determining size and/or location information associated with one or more physiological characteristics.

Fig. 18B shows a light pattern 1010 scanned over the eye 1002A. In the example shown, the light pattern has a "V" shape that may include an "alpha" 1010a portion and a "beta" portion 1010 b. The "V" shape is described in more detail above with respect to at least fig. 10A-10B. An example light intensity pattern 1814 is shown that indicates a measure of signal or current from a photodetector that receives light reflected from the eye 1002A during a scan. The example light intensity pattern 1814 may be associated with one of a plurality of photodetectors positioned with respect to the eye 1002A. Fig. 11 shows an example of a light detector positioned with respect to an eye.

At the time or movable reflector position represented by fig. 18B, the "alpha" portion 1010a of the light pattern 1010 is swept at the interface or boundary 1812 between the sclera and the iris. This intersection 1812 may show a reduction in reflected light received by the light detector associated with pattern 1814 as light passes from the highly reflective white sclera to the darker, less reflective iris. The reflected light may result from diffuse reflection of the light pattern 1010, and the pattern 1814 may thus indicate a gradual decrease in intensity. Thus, the light intensity pattern 1814 indicates this interface 1812 as a reduction in signal relative to the portion 1816 of the light intensity pattern 1814.

The display system may determine that the reduction in signal corresponds to a physiological interface 1812 between the sclera and the iris using techniques described herein, such as a machine learning model or stored information. For example, the display system may use a threshold number of light intensity patterns to effect the determination. The display system may then determine a location associated with the interface 1812. For example, the display system may identify the time mapped to portion 1816. In this example, the display system may determine the position based on the speed at which the movable reflector is adjusted. As another example, the display system may identify a movable reflector position associated with the portion 1816. Based on the information identifying the position of the movable reflector, the display system can determine the location at which the light pattern 1010 is incident and the location of the portion 1816 that caused the light intensity pattern 1814.

Fig. 18C shows the light pattern 1010 further scanned over the eye 1002A. In this example, the "alpha" portion 1010a is swept through the intersection 1822 between the iris and the pupil. Similar to above, the intersection may show a reduction in light reaching the photodetector associated with the light intensity pattern 1814 as light passes from the more reflective iris to the darker, less reflective iris. Thus, a decrease in signal is indicated at the portion 1824 of the light intensity pattern 1814. The display system may thus determine the location associated with this interface 1822 as described above.

Fig. 18D shows the light pattern 1010 being further scanned until an "alpha" portion 1010a is scanned over another interface 1832 between the pupil and the iris. The interface 1832 represents the rightmost interface between the pupil and the iris. In contrast, the interface 1822 depicted in fig. 18C represents the leftmost interface between the pupil and the iris. Similar to above, the display system may determine a location associated with the interface 1832. However, in the illustrated light intensity pattern 1814, the peak 1834 caused by the hot spot coincides with the interface 1832. It will be appreciated that when the light pattern 1010 is scanned over the interface 1834, the bright spots provide a large amount of reflected light (e.g., specularly reflected light), which may obscure detection of the interface 1834 using the illustrated light intensity pattern 1814. However, as discussed herein, it is preferred to use multiple photodetectors that provide multiple light intensity patterns, and at least some of these other photodetectors would be expected to record differences in reflected light intensity due to variations in reflectivity at the interface 1832. For example, these other light intensity patterns (not shown) may show the reverse (reverse) of

portions

1824 and 1816 of the light intensity pattern 1814 and may be used to determine the location of the interface 1832.

Based on interface 1822 shown in fig. 18C and interface 1832 shown in fig. 18D, the display system may determine the size associated with the pupil. For example, the size may represent a size along a horizontal axis. In this example, the size may thus represent the length of the pupil along the horizontal axis, e.g. the width or diameter of the pupil. The display system may additionally determine the location of the pupil. For example, the display system may calculate the center of mass of the pupil (e.g., the midpoint of the pupil width) as determined based on

interfaces

1822, 1832.

It should be understood that the interface or boundary between the iris and the sclera, and the size of the iris, may be determined as discussed above for the interface between the iris and the pupil and the size of the pupil. For example, the location of the left and right interfaces of the iris may be determined based on the detected reduction in reflected light and the reflected light level being higher than the lower reflected light level of the pupil.

While the above description identifies that the display system may determine the interface (e.g., interface 1812) before the scan is completed, in some embodiments, the display system may determine the interface when the scan is completed. For example, the display system may obtain a light intensity pattern and determine size and/or position information based on the light intensity pattern. In this example, machine learning techniques may be utilized. For example, one or more machine learning models may be trained to identify (e.g., label) physiological features based on light intensity patterns.

Estimating eye velocity

In some embodiments, the display system may determine the velocity of the user's eyes. For example, the display system may determine the rotational speed. The rotation speed may be used by the display system for different purposes, such as identifying the occurrence of a saccade or estimating the extent to which the eye will rotate during a saccade. As will be described, the display system may determine a rotational speed (e.g., saccade velocity) based on comparing the motion of one or more physiological characteristics or differences between successive eye poses.

The display system may determine the rotation speed based on continuously determined differences in eye posture. For example, at a first time, the display system may perform a scan of the user's eyes. In this example, the display system may associate a first eye pose determined based on the scan with the first time. At a second time, the display system may perform a subsequent scan of the user's eyes. The display system may then associate the second eye pose with the second time. One or more measurements of an eye pose difference between the first eye pose and the second eye pose may be determined. Example measurements may include adjusting an optical or visual axis between a first eye pose and a second pose. The display system may then determine a rotational speed using the determined difference in optical or visual axis position and the time difference between the first time and the second time.

The display system may also determine a rotational speed based on the movement of the one or more physiological characteristics. For example, at a first time, the display system may determine a first location of a physiological feature (e.g., an interface between an iris and a pupil). Subsequently, at a second time, the display system may determine a second location of the physiological characteristic. The extent to which the physiological characteristic has moved, for example, along one or more axes, can be identified. The display system may then determine a rotational speed based on the difference in the determined locations of the physiological characteristics and the difference between the first time and the second time.

A saccade may be understood to mean a rapid movement of the eye between two or more fixation phases. During the occurrence of a glance, the user may reduce visibility between the two fixations. In some embodiments, the display system may adjust the presentation of the virtual content to take advantage of this reduced visibility. To identify the occurrence of a saccade, the display system may determine whether the rotational speed of the eyes exceeds a threshold. Due to the techniques described herein, the display system may advantageously scan the eye at a rate high enough to detect saccades (e.g., 1kHz, 10kHz, etc.). Thus, the occurrence of a glance can be determined. This determination may be used to affect depth plane switching in a multi-depth flat display, as discussed, for example, in U.S. patent application publication No.2017/0276948, published on 9/28/2017, the entire contents of which are incorporated herein by reference.

Further, it should be understood that the degree to which the eye will rotate during a saccade may be based on the initial rotational speed of the eye (saccade velocity). For example, it will be appreciated that the initial saccadic velocity of the eye, the angle of eye movement and the final orientation of the eye after saccades are related. Thus, if the initial velocity and direction are known, the display system may estimate the final position at which the user will be gazing at the completion of the sweep.

It should be appreciated that different users may have different associated saccade velocities. The display system may advantageously use machine learning techniques to generate a model associated with a particular user's saccadic velocities. For example, the display system may identify an initial velocity and direction associated with the glance. In this example, the display system may then identify the degree of eye rotation at the completion of the saccade (e.g., by constantly scanning the eye and determining a gesture as described herein). Based on this information, the display system may train or otherwise update an existing machine learning model. For example, a machine learning model may learn an accurate correlation between saccadic velocity, direction, and end point of a particular user's eyes.

Being able to estimate the endpoint may allow a final post-saccade gesture to be determined. As discussed herein, the display system may use this estimated pose when presenting virtual content to the user. In some embodiments, the estimated pose may be used to verify and/or determine a confidence level for the pose determined using the light scan and light intensity pattern based pose discussed herein.

Other embodiments

The aspects, embodiments, implementations, or features of the described embodiments may be used alone or in any combination. Aspects of the described embodiments may be implemented in software, hardware, or a combination of hardware and software. The described embodiments can also be implemented as computer readable code on a computer readable medium for controlling a manufacturing operation or computer readable code on a computer readable medium for controlling a manufacturing line. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium include read-only memory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tape, and optical data storage devices. The computer readable medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the described embodiments. Thus, the foregoing descriptions of specific embodiments have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. Many modifications and variations may be apparent to those of ordinary skill in the art in light of the above teachings.

It will be understood that each of the processes, methods, and algorithms described herein and/or illustrated in the accompanying figures can be embodied in, and automated in whole or in part by: code modules executed by one or more physical computing systems, hardware computer processors, application specific circuits, and/or electronic hardware configured to execute specific and specific computer instructions. For example, a computing system may include a general purpose computer (e.g., a server) or a special purpose computer, special purpose circuitry, etc., which is programmed with specific computer instructions. The code modules may be compiled and linked into executable programs, installed in dynamically linked libraries, or written in an interpreted programming language. In some embodiments, certain operations and methods may be performed by circuitry that is dedicated to a given function.

Furthermore, certain embodiments of the disclosed functionality are complex enough, mathematically, computationally, or technically, to require either dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) in order to perform the functionality (e.g., as dictated by the amount of computation or complexity involved) or to provide the results in substantially real time. For example, video may comprise many frames, each having millions of pixels, and require specially programmed computer hardware to process the video data to provide the required image processing tasks or applications in a commercially reasonable time.

The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage including hard drives, solid state memory, Random Access Memory (RAM), Read Only Memory (ROM), optical disks, volatile or non-volatile storage, combinations of the same or similar elements, and the like. In some embodiments, the non-transitory computer-readable medium may be part of one or more of a local processing and data module (140), a remote processing module (150), and a remote data store (160). The methods and modules (or data) may also be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based and wire/cable-based media, which may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

Any process, block, state, step, or function in the flowcharts described herein and/or shown in the figures should be understood as potentially representing a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified function(s) (e.g., logical or arithmetic function) or step in the process. Various processes, blocks, states, steps or functions may be performed with the illustrative examples provided herein to perform the following operations: combinations, rearrangements, additions, deletions, modifications or other changes. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular order, and the blocks, steps, or states associated therewith may be performed in other suitable sequences, e.g., in serial, parallel, or in some other manner. Tasks or events can be added to, or removed from, the disclosed example embodiments. Moreover, the separation of various system components in the embodiments described herein is for illustrative purposes, and should not be construed as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products.

In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. For example, while advantageously used with AR displays that provide images across multiple depth planes, augmented reality content disclosed herein may also be displayed by systems that provide images on a single depth plane.

Indeed, it will be appreciated that the systems and methods of the present disclosure each have several inventive aspects, no single one of which is solely responsible for or is essential to the desirable characteristics disclosed herein. The various features and processes described above may be used independently of one another or may be used in various combinations. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure.

Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. No single feature or group of features is essential or essential to each embodiment.

It will be understood that terms such as "may," "can," "e.g.," as used herein, are generally intended to convey that certain embodiments include, but other embodiments do not include, certain features, elements, and/or steps unless specifically stated otherwise, or otherwise understood in the context. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments and that one or more embodiments necessarily include logic for deciding, with or without programmer input or prompting, whether such features, elements, and/or steps are included or are to be performed in any particular embodiment. The terms "comprising," "including," "having," and the like, are synonymous and are used inclusively, in an open-ended fashion, and do not exclude other elements, features, acts, operations, and the like. Furthermore, the term "or" is used in its inclusive sense (and not exclusive sense), so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more" or "at least one" unless specified otherwise. Similarly, while operations may be shown in the drawings in a particular order, it should be recognized that these operations need not be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Further, the figures may schematically illustrate one or more example processes in flow chart form. However, other operations not shown may be incorporated into the example methods and processes shown schematically. For example, one or more additional operations may be before, after, concurrently with, or during any of the illustrated operations. Additionally, in other embodiments, the operations may be rearranged or ordered. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims

1. A display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

one or more processors configured to perform operations comprising:

causing an adjustment of an orientation of the movable reflector to cause the reflected light to be scanned over the eye;

2. The display system of claim 1, wherein the light source is a diode.

3. The display system of claim 2, wherein the diode is a vertical cavity surface emitting laser.

4. The display system of claim 1, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light rays that span an area of the eye.

5. The display system of claim 1, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light beams.

6. The display system of claim 1, wherein the movable reflector comprises a plurality of diffraction gratings, each diffraction grating configured to form a different light pattern for scanning over the eye.

7. The display system of claim 1, wherein the movable reflector is a micro-electromechanical system (MEMS) mirror.

8. The display system of claim 1, wherein the light detector is a photodiode, and wherein each light intensity pattern represents a plot of current versus position information associated with the position of the movable reflector.

9. The display system of claim 8, wherein the diffraction grating is located on or forms part of a MEMS mirror, and wherein the position information indicates an orientation of the MEMS mirror, the MEMS being adjustable by the display system.

10. The display system of claim 1, wherein the light source is one of two light sources configured to output light to the movable reflector, wherein each of the light sources is configured to form a respective portion of the light pattern.

11. The display system of claim 1, wherein the light detector is a photodiode, and wherein each light intensity pattern represents a plot of current versus time.

12. The display system of claim 1, wherein the light pattern defines a V-shape extending from a lower portion of the eye to an upper portion of the eye.

13. The display system of claim 1, wherein the light forming the light pattern comprises polychromatic light.

14. The display system of claim 13, wherein the light pattern comprises two portions extending in different directions.

15. The display system of claim 14, wherein each of the two portions is formed from a different color of light.

16. The display system of claim 14, wherein the two portions are configured to extend across a vertical axis of the eye, wherein the two portions extend in opposite directions along a horizontal axis to form a V-shape.

17. The display system of claim 1, wherein the light pattern comprises a plurality of consecutive rows of light.

18. The display system of claim 17, wherein different rows of light comprise light beams having different amounts of divergence.

19. The display system of claim 18, wherein a row of light comprises a converging light beam, wherein another of the rows of light comprises a collimated light beam.

20. The display system of claim 18, wherein the row of light comprises a diverging beam of light.

21. The display system of claim 17, wherein the row of light defines an angle of less than 90 ° relative to a horizontal axis of the eye.

22. The display system of claim 1, wherein the position of the light detector defines a corner of a rectangle about the eye.

23. The display system of claim 1, wherein the light detector defines a linear array of light detectors.

24. The display system of claim 1, wherein the operations further comprise: such that the light pattern is continuously scanned over an axis between the first portion of the eye and the second portion of the eye.

25. The display system of claim 24, wherein the axis is a horizontal axis of the eye such that the first portion is a leftmost portion or a rightmost portion of the eye and the second portion is the other of the leftmost portion or the rightmost portion of the eye.

26. The display system of claim 1, wherein determining the eye pose comprises:

27. The display system of claim 1, wherein determining the eye pose comprises:

identifying the eye pose based on the comparison.

28. The display system of claim 26, wherein the light detector is a photodiode, wherein comparing the obtained light intensity pattern to the stored light intensity pattern is based on comparing locations of peaks and/or valleys of current, and wherein the locations are indicative of locations of the optical pattern on the eye.

29. The display system of claim 1, wherein the operations further comprise:

determining an interpupillary distance of the user;

Scanning the light pattern over the eye at the scan distance.

30. The display system of claim 1, wherein the operations further comprise detecting one or both of an iris and a pupil of the eye based on the light intensity pattern.

31. The display system of claim 30, wherein detecting one or both of the iris and the pupil of the eye comprises: determining a size of one or both of the iris and the pupil of the eye.

32. The display system of claim 30, wherein detecting one or both of the iris and the pupil of the eye comprises: determining a location of one or both of the iris and the pupil of the eye.

33. The display system of claim 1, wherein the operations further comprise: determining a saccadic velocity of the eye.

34. The display system of claim 1, further comprising: a waveguide comprising out-coupling optical elements configured to output light to the user's eye to form the virtual content.

35. The display system of claim 29, wherein the waveguide is one waveguide in a stack of waveguides, wherein some waveguides in the stack have outcoupling optical elements configured to output light having a different amount of wavelength divergence than outcoupling optical elements of other waveguides in the stack, wherein the different amounts of wavefront divergence correspond to different depth planes.

36. A method implemented by a display system of one or more processors configured to present virtual content to a user based at least in part on eye gestures of the user's eyes, wherein the method comprises:

37. The method of claim 36, wherein adjusting the position of the light pattern comprises: moving a movable mirror to cause the light pattern to move along an axis from a first portion of the eye to a second portion of the eye.

38. The method of claim 37, wherein the movable reflector comprises a diffraction grating, wherein the diffraction grating is configured to convert an incident light beam from the light source into a light pattern comprising a plurality of light beams.

39. The method of claim 37, wherein moving the movable reflector comprises rotating a micro-electromechanical systems (MEMS) mirror on which the diffraction grating is located.

40. The method of claim 37, wherein the first portion represents a terminal end of the iris and the second portion represents an opposite terminal end of the iris along the axis.

41. The method of claim 37, wherein the axis is a horizontal axis.

42. The method of claim 36, wherein the light pattern extends along a vertical axis from a lower portion of the eye to an upper portion of the eye.

43. The method of claim 42, wherein the light pattern comprises two portions, each portion extending along a vertical axis, and wherein the two portions extend in opposite directions along a horizontal direction to form a V-shape.

44. The method of claim 36, wherein determining an eye pose comprises:

45. The method of claim 36, wherein determining an eye pose comprises:

identifying the eye pose based on the comparison.

46. A method as claimed in claim 45, wherein comparing the obtained light intensity pattern with the stored light intensity pattern is based on comparing the position of peaks and/or troughs in the light intensity pattern.

47. A non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

48. The computer storage medium of claim 47, wherein the operations further comprise:

49. The computer storage medium of claim 48, wherein the orientation of the diffraction grating is adjusted such that the light pattern moves from a first portion of the eye to a second portion of the eye.

50. The computer storage medium of claim 49, wherein the first portion represents a distal end of the iris and the second portion represents an opposite distal end of the iris.

51. The computer storage medium of claim 47, wherein the light pattern extends along a vertical axis from a lower portion of the eye to an upper portion of the eye.

52. The computer storage medium of claim 51, wherein the light portion comprises two portions, each portion extending along a vertical axis over the eye, and wherein the two portions extend along a horizontal axis in opposite directions.

53. The computer storage medium of claim 47, wherein adjusting the orientation of the diffraction grating comprises: controlling rotation of a micro-electro-mechanical system (MEMS) mirror on which the diffraction grating is located.

54. The computer storage medium of claim 47, wherein determining an eye pose comprises:

55. The computer storage medium of claim 47, wherein determining an eye pose comprises:

identifying the eye pose based on the comparison.

56. The computer storage medium of claim 55, wherein comparing the obtained light intensity pattern to the stored light intensity pattern is based on comparing locations of peaks and/or valleys.

57. A display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

one or more processors configured to perform operations comprising:

58. The display system of claim 57, wherein the physiological characteristic is a pupil of the eye.

59. The display system of claim 58, wherein the operations further comprise:

60. The display system of claim 59, wherein determining the first interface is based on a location of a peak and/or a valley in the light intensity pattern.

61. The display system of claim 59, wherein the operations further comprise:

62. The display system of claim 61, wherein the size of the pupil is determined based on the first interface and the second interface.

63. The display system of claim 61, wherein the physiological characteristic is the pupil, and wherein the position of the pupil is determined based on a center of the pupil, the center identified based on the first and second interfaces.

64. The display system of claim 57, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

65. A method implemented by a display system of one or more processors configured to present virtual content to a user based at least in part on eye gestures of the user's eyes, wherein the method comprises:

66. The method of claim 65, wherein the physiological characteristic is a pupil of the eye.

67. The method of claim 66, further comprising:

68. The method of claim 67, wherein determining the first interface is based on a location of a peak and/or a valley in the light intensity pattern.

69. The method of claim 68, further comprising:

70. The method of claim 69, wherein the physiological characteristic is the pupil, and wherein a size of the pupil is based on the first interface and the second interface.

71. The method of claim 69, wherein the physiological characteristic is the pupil, and wherein the location of the pupil is based on a center of the pupil, the center identified based on the first interface and the second interface.

72. The method of claim 65, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

73. A non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

74. The computer storage medium of claim 73, wherein the operations further comprise:

75. The computer storage medium of claim 74, wherein determining the interface is based on locations of peaks and/or valleys in the light intensity pattern.

76. The computer storage medium of claim 74, wherein the operations further comprise:

77. The computer storage medium of claim 76, wherein the physiological characteristic is the pupil, and wherein a size of the pupil is determined based on the first interface and the second interface.

78. The computer storage medium of claim 76, wherein the physiological characteristic is the pupil, and wherein the location of the pupil is based on a center of the pupil, the center identified based on the first interface and the second interface.

79. The computer storage medium of claim 73, wherein the physiological characteristic is an interface between an iris and a pupil of the eye, and wherein the display system determines a location of the interface.

80. A display system configured to present virtual content to a user, the display system comprising:

a light source configured to output light;

one or more processors configured to perform operations comprising:

determining a rotation speed of the eye based on the light intensity pattern.

81. The display system of claim 80, wherein determining the rotational speed of the eye comprises: determining a saccadic velocity of the eye.

82. The display system of claim 81, wherein the operations further comprise: predicting a pose of the eye based on the saccade velocity.

83. A method implemented by a display system of one or more processors configured to present virtual content to a user based at least in part on eye gestures of the user's eyes, wherein the method comprises:

determining a rotation speed of the eye based on the light intensity pattern.

84. A non-transitory computer storage medium storing instructions that, when executed by a display system of one or more processors, cause the one or more processors to perform operations comprising:

determining a rotation speed of the eye based on the light intensity pattern.