JP2020514926A - Depth-based foveated rendering for display systems - Google Patents

Abstract

A method and system for depth-based foveated rendering in a display system is disclosed. The display system may be an augmented reality display system configured to provide virtual content on multiple depth planes using different wavefront divergence. Some embodiments include monitoring the eye orientation of a user of the display system based on the detected sensor information. The fixation point is determined based on the eye orientation, and the fixation point represents a three-dimensional place with respect to the visual field. The location information of the virtual object for presentation is acquired, and the location information indicates the three-dimensional position of the virtual object. The resolution of the at least one virtual object is adjusted based on the proximity of the at least one virtual object and the fixation point. The virtual object is presented to the user by the display system and the at least one virtual object is rendered according to the adjusted resolution.

Description

(Priority claim)
This application is US provisional application No. 62 / 644,365 (filing date March 16, 2018), US provisional application No. 62 / 475,012 (filing date March 22, 2017), US provisional application No. 62. / 486,407 (filing date April 17, 2017) and US provisional application No. 62 / 539,934 (filing date August 1, 2017). The above patent applications are hereby incorporated by reference in their entireties for all purposes.

(Incorporated by reference)
This application is incorporated by reference in its entirety for each of the following patent applications and publications: US Application No. 14 / 555,585 (filing date November 27, 2014), US Publication No. 2015/0205126 as 2015. Published on July 23, 2014; US application No. 14 / 690,401 (filed on April 18, 2015), published on October 22, 2015 as US publication No. 2015/0302652; US application No. 14 / No. 212,961 (filing date March 14, 2014), currently issued on August 16, 2016, US Pat. No. 9,417,452; US application No. 14 / 331,218 (filing date July 2014) 14), published on October 29, 2015 as US Publication No. 2015/0309263; US Application No. 15 / 902,927 (filing date February 22, 2018); US Provisional Application No. 62 / 475,012. No. (filing date March 22, 2017); and US provisional application No. 62 / 539,934 (filing date August 1, 2017).

  The present disclosure relates to display systems, including augmented reality imaging and visualization systems.

  Modern computing and display technologies are driving the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or parts thereof are real. Presented to the user in a manner that can be seen or so perceived. Virtual reality or “VR” scenarios typically involve the presentation of digital or virtual image information without transparency to other real world visual inputs, while augmented reality or “AR” scenarios , Typically with the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. A mixed reality or "MR" scenario is a type of AR scenario that typically involves virtual objects integrated into and responsive to the natural world. For example, an MR scenario may include AR image content that appears to be blocked by an object in the real world, or otherwise perceived as interacting with it.

  Referring to FIG. 1, an augmented reality scene 10 is depicted. Users of AR technology see a real-world park setting 20, featuring people, trees, buildings in the background, and concrete platforms 30. The user also "sees" the virtual image 40 of the robot image 40 standing on the real-world platform 30 and the flying cartoon avatar character 50, which looks like a bumblebee anthropomorphic. Perceive. These elements 50, 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex and facilitates the comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements. Have difficulty.

  The systems and methods disclosed herein address various issues associated with AR or VR technology.

  According to some embodiments, a system includes one or more processors and one or more computers storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. And a storage medium. The operations include monitoring eye movements of the user based on information detected via the one or more sensors. The fixation point that the user's eye is fixing is determined based on the eye movement, and the fixation point is a three-dimensional place in the user's visual field. The act includes obtaining location information associated with one or more virtual objects for presentation to a user, where the location information indicates a three-dimensional position of the virtual object. The act of also includes adjusting the resolution of the at least one virtual object based at least in part on the proximity of the fixation point to the at least one virtual object. The act of includes also providing to the user, via the display, a presentation of the virtual object, the at least one virtual object being rendered according to the adjusted resolution.

  According to some embodiments, a display system is configured to present virtual content to a user, a display device, one or more processors, and instructions for causing the system to perform operations when executed by the system. And one or more computer storage media for storing The act includes monitoring information associated with the user's eye movements. A fixation point within the display frustum of the display device is determined based on the monitored information, the fixation point indicating a three-dimensional location being fixed by the user's eye. The act of also includes presenting the virtual content to a three-dimensional location within the display frustum based on the determined fixation point, the virtual content providing a resolution based on the proximity of the virtual content from the fixation point. Adjusted.

  According to some other embodiments, the method includes monitoring the eye orientation of the user of the display device based on the information detected via the one or more sensors. The fixation point that the user's eye is fixing is determined based on the eye orientation, and the fixation point is a three-dimensional place in the user's visual field. Place information associated with one or more virtual objects for presentation to a user is obtained, where the place information indicates a three-dimensional position of the virtual object. The resolution of the at least one virtual object is adjusted, at least in part, based on the proximity of the at least one virtual object and the fixation point. The method also includes providing to the user via the display a presentation of the virtual object, the at least one virtual object being rendered according to the adjusted resolution.

  According to some embodiments, a display system comprises a frame configured to be mounted on a user's head and a light modulation system configured to output light and form an image. One or more waveguides mounted to the frame and configured to receive light from the light modulation system and output the light across the surface of the one or more waveguides. The system also includes one or more processors and one or more computer storage media that, when executed by the one or more processors, store instructions that cause the one or more processors to perform various operations. Prepare The act includes determining the amount of light reaching the retina of the user's eye and adjusting the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina. ..

  According to some other embodiments, a display system comprises one or more processors and one or more computer storage media storing instructions. When instructions are executed by one or more processors, they cause the one or more processors to perform various operations. The act of determining the amount of light reaching the retina of the user's eye of the display system and adjusting the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina. Including and

  According to some embodiments the method is implemented by a display system comprising one or more processors and a head mountable display. The method determines the amount of light reaching the retina of the user's eye of the display system and adjusts the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina. And steps.

  According to some other embodiments, a display system is configured to be mounted on a user's head, a frame, a light modulation system, one or more waveguides, and one or more A processor and one or more computer storage media storing instructions are included. The light modulation system is configured to output light and form an image. The one or more waveguides are mounted to the frame and are configured to receive light from the light modulation system and output the light across the surface of the one or more waveguides. One or more computer storage media store instructions that, when executed by one or more processors, cause the one or more processors to perform various operations. The operation includes adjusting the resolution of the primary color image forming the virtual content based on the proximity of the virtual content from the user's perspective and the color of the primary color image. At least one of the primary color images has a different resolution from the primary color image of another color.

  According to yet another embodiment, a display system comprises one or more processors and one or more computer storage media storing instructions. When instructions are executed by one or more processors, they cause the one or more processors to perform various operations. The act includes adjusting the resolution of the primary color images forming the virtual content based on the proximity of the virtual content from the user's perspective and the color of the primary color image, at least one of the primary color images comprising: The resolution is different from the primary color image of another color.

  According to some other embodiments, the method is implemented by a display system comprising one or more processors and a head mountable display. The method includes adjusting the resolution of the primary color images forming the virtual content based on the proximity of the virtual content from the user fixation point and the color of the primary color image, at least one of the primary color images being , The resolution is different from the primary color image of another color.

  According to yet another embodiment, a display system is in communication with an image source, a viewing assembly, and an image source comprising a spatial light modulator for providing a first image stream and a second image stream. One or more processors and one or more computer storage media that store instructions that, when executed by one or more processors, cause the one or more processors to perform various operations. The viewing assembly comprises light guiding optics for receiving the first and second image streams from the image source and outputting the first and second image streams to a user. Various operations performed by the one or more processors include causing an image source to output a first image stream to a viewing assembly, wherein the image formed by the first image stream is a first pixel. Having a density, and causing the image source to output a second image stream to a viewing assembly. The image formed by the second image stream has a second pixel density that is greater than the first pixel density and corresponds to the portion of the image provided by the first image stream. The image formed by the second image stream covers the corresponding portion of the field of view provided by the first image stream.

  According to some embodiments, the wearable display system may include an afocal magnifying lens with circular polarization handedness dependent magnification. The afocal magnifying lens exhibits positive refractive power for the first fixed focal length lens element and the first handedness of the incident circularly polarized light and negative power for the second handedness of the incident circularly polarized light. It may include a first geometric phase lens and a second geometric phase lens that exhibit refractive power.

  According to some other embodiments, an optical subsystem for a wearable image projector may include a polarization selective reflector and a set of four lens elements centered on the polarization selective reflector. ..

  A display system for projecting an image onto a user's eye, according to some other embodiments, may include an eyepiece. The eyepiece may include a waveguide and an incoupling grating that is optically coupled to the waveguide. The display system may further include a first image source configured to project a first light beam associated with the first image stream. The first image stream may have a first field of view and may be incident on the first surface of the incoupling grating. A portion of the first light beam may be coupled into the waveguide by an incoupling grating to position the first image stream in a fixed position relative to the user's eye. The display system may further include a second image source configured to project a second light beam associated with the second image stream. The second image stream may have a second field of view that is narrower than the first field of view. The display system may further include a scanning mirror for receiving the second light beam such that the second light beam is incident on the second surface of the incoupling grating opposite the first surface thereof. , May be configured to reflect. A portion of the second light beam may be coupled into the waveguide by an incoupling grating. The display system may further include an eye gaze tracker configured to detect movement of the user's eye and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

  According to some other embodiments, a display system for projecting an image on a user's eye may include an eyepiece. The eyepiece may include a waveguide and an incoupling grating that is optically coupled to the waveguide. The display system further includes a first light beam associated with the first image stream at the first polarization and a second light beam associated with the second image stream at a second polarization different from the first polarization. An image source may be included that is configured to project the beam. The first image stream may have a first field of view and the second image stream may have a second field of view that is narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system is further configured to receive the first light beam, reflect it along the first optical path, receive the second light beam, and transmit it along the second optical path. A polarizing beam splitter may be included. The display system is further positioned along the first optical path to receive and reflect the first light beam such that the first light beam is incident on the first surface of the incoupling grating. The first optical reflector may be included. A portion of the first light beam may be coupled into the waveguide by an incoupling grating to position the first image stream in a fixed position relative to the user's eye. The display system further includes a scan mirror disposed along the second optical path and configured to receive and reflect the second light beam, and a second optical path downstream of the scan mirror. A second optical reflector that is positioned. The second optical reflector receives and reflects the second light beam such that the second light beam is incident on the second surface of the incoupling grating opposite the first surface thereof. May be configured as. A portion of the second light beam may be coupled into the waveguide by an incoupling grating. The display system may further include an eye gaze tracker configured to detect movement of the user's eye and a control circuit in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

  According to some other embodiments, a display system for projecting an image onto a user's eye comprises a waveguide and a first light beam associated with a first image stream in a first polarization. An image source configured to project a second light stream associated with a second image stream in a second polarization different from the first polarization. The first image stream may have a first field of view and the second image stream has a second field of view that is narrower than the first field of view. The first light beam and the second light beam may be multiplexed. The display system is further configured to receive the first light beam, reflect it along the first optical path, receive the second light beam, and transmit it along the second optical path. A polarizing beam splitter may be included. The display system may further include a first incoupling prism positioned along the first optical path and adjacent the first surface of the waveguide. The first incoupling prism may also be configured to couple a portion of the first light beam into the waveguide to position the first image stream in a fixed position relative to the user's eye. Good. The display system may further include a scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam. The display system further includes a second incoupling positioned adjacent the second surface of the waveguide opposite the first surface of the waveguide along a second optical path downstream of the scanning mirror. It may include a prism. The second incoupling prism may be configured to couple a portion of the second light beam into the waveguide. The display system may further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

  According to one embodiment, a display system for projecting an image onto a user's eye includes an image source. The image source includes a first light beam associated with a first image stream in a first polarization and a second light beam associated with a second image stream in a second polarization different from the first polarization. And can be configured to project. The first image stream may have a first field of view and the second image stream may have a second field of view that is narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system can further include a polarizing beamsplitter. The polarizing beam splitter receives the first light beam and reflects it toward the viewing assembly along a first optical path to position the first image stream in a fixed position relative to the user's eye. It may be configured to receive two light beams and transmit them along the second optical path. The display system may further include a scanning mirror disposed along the second optical path and configured to receive the second light beam and reflect it toward the viewing assembly. The display system can further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuit may be configured to position the scanning mirror such that the position of the second image stream is moved according to the detected movement of the user's eye.

  According to another embodiment, a display system for projecting an image onto a user's eye includes an image source. The image source may be configured to project a first light beam associated with the first image stream and a second light beam associated with the second image stream. The first image stream may have a first field of view and the second image stream may have a second field of view that is narrower than the first field of view. The first light beam and the second light beam can be multiplexed. The display system is further configured to receive the first light beam and the second light beam and project them toward the viewing assembly to project the first image stream and the second image stream. A scanning mirror can be included. The display system can further include an eye gaze tracker configured to detect movement of the user's eye, and control circuitry in communication with the eye gaze tracker and the scanning mirror. The control circuit can be configured to position the scanning mirror such that the position of the first image stream and the position of the second image stream are moved according to the detected movement of the user's eye. The display system can further include switchable optical elements disposed in the optical paths of the first light beam and the second light beam. The switchable optical element is switched to a first state for the first light beam such that the first light beam is angularly expanded by the first angular expansion factor, and the second light beam is It can be configured to be switched to a second state for the second light beam so as to be angularly amplified by a second angular magnification that is less than the first angular magnification.

FIG. 1 illustrates a view of an augmented reality (AR) user through an AR device.

FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.

3A-3C illustrate the relationship between radius of curvature and radius of focus.

FIG. 4A illustrates a representation of the accumodation-convergence / divergence movement response of the human visual system.

FIG. 4B illustrates an example of different accommodation and vergence / divergence movements of the eyes of a pair of users.

FIG. 4C illustrates an example of a top and bottom representation of a user viewing content via a display system.

FIG. 4D illustrates another example of a top and bottom representation of a user viewing content via a display system.

FIG. 5 illustrates aspects of an approach for simulating a three-dimensional image by modifying the wavefront divergence.

FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user.

FIG. 7 illustrates an embodiment of the exit beam output by the waveguide.

FIG. 8 illustrates an example of stacked waveguide assemblies, each depth plane containing an image formed using a plurality of different primary colors.

FIG. 9A illustrates a cross-sectional side view of an example of a set of stacked waveguides, each including an incoupling optical element.

FIG. 9B illustrates a perspective view of the multiple stacked waveguide embodiment of FIG. 9A.

FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B.

FIG. 9D illustrates an example of a wearable display system.

FIG. 10A illustrates an example of a top-bottom representation of a user viewing content via a display system.

FIG. 10B illustrates another example of a top-bottom representation of a user viewing content via a display system.

FIG. 10C illustrates yet another example of a top-bottom representation of a user viewing content via a display system.

FIG. 10D is a block diagram of an exemplary display system.

FIG. 11A1 illustrates an example of a top-down representation of adjustments at resolutions within different resolution adjustment zones based on 3D fixation tracking.

FIG. 11A2 illustrates an example of a top and bottom representation of the resolution adjustment zone at different times as the size and number of zones change.

FIG. 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone of FIG. 11A1.

FIG. 11C illustrates another example of a configuration for the resolution adjustment zone.

FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

11E illustrates another example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C.

12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to proximity to a 3D fixation point. 12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to proximity to a 3D fixation point. 12A-12C show schematic diagrams of an example of a process for adjusting the resolution of content according to proximity with a three-dimensional fixation point.

FIG. 13 illustrates an example of a representation of a user viewing a plurality of virtual objects that are aligned with the user's line of sight.

FIG. 14 is a schematic diagram of an example of a process for adjusting virtual content based on angular proximity to a user's line of sight.

FIG. 15 illustrates an example of a representation of the retina of the user's eye.

FIG. 16 schematically illustrates an example of trans-retinal resolution and rod and cone density of FIG.

FIG. 17 diagrammatically illustrates an example of the relationship between pupil size and the amount of light incident on the user's eye.

FIG. 18 is a schematic diagram of an example of a process for adjusting virtual content based on the amount of light incident on a user's eye.

FIG. 19 diagrammatically illustrates an example of the change in resolution detectable by the user's eye as the amount of light incident on the eye changes.

FIG. 20 schematically illustrates an example of the difference in the sensitivity of the eye to different colors of light at different levels of illumination.

FIG. 21 is a schematic diagram of an embodiment of a process for adjusting virtual content formed using multiple primary color images, where resolution adjustment is based on the colors of the primary color images.

22A-22C illustrate examples of contrast sensitivity that change as the amount of light incident on a user's eye decreases.

FIG. 23 illustrates an example representation of the optic nerve and peripheral blind spots of the user's eye.

FIG. 24 shows an exemplary monocular field of view for the human eye.

FIG. 25A illustrates an exemplary wearable display device configured to provide virtual content to a user.

FIG. 25B is a block diagram depicting an augmented reality system.

FIG. 25C schematically illustrates a light path within a viewing optics assembly (VOA) that may be used to present a digital or virtual image to a viewer.

26A-26D illustrate exemplary rendering viewpoints that will be used and the light fields that will be produced in the AR system for every two exemplary eye orientations. 26A-26D illustrate exemplary rendering viewpoints that will be used and the light fields that will be produced in the AR system for every two exemplary eye orientations. 26A-26D illustrate exemplary rendering viewpoints that will be used and the light fields that will be produced in the AR system for every two exemplary eye orientations. 26A-26D illustrate exemplary rendering viewpoints that will be used and the light fields that will be produced in the AR system for every two exemplary eye orientations.

26E-26F schematically illustrate exemplary configurations of images that may be presented to a user. 26E-26F schematically illustrate exemplary configurations of images that may be presented to a user.

26G-26H schematically illustrate exemplary configurations of images that may be presented to a user. 26G-26H schematically illustrate exemplary configurations of images that may be presented to a user.

FIG. 27 illustrates a visual field and an ocular visual field as shown in FIG. 24 overlaid on one of the displays in a wearable display device as shown in FIG.

28A-28B illustrate some of the principles described in FIGS. 26A-26D. 28A-28B illustrate some of the principles described in FIGS. 26A-26D.

28C-28D illustrate some example images that may be presented to a user. 28C-28D illustrate some example images that may be presented to a user.

FIG. 28E illustrates an exemplary high FOV low resolution image frame.

FIG. 28F illustrates an exemplary low FOV high resolution image frame.

FIG. 29A shows a simplified block diagram of a display system.

FIG. 29B schematically illustrates a cross-sectional view of an augmented reality (AR) system.

30A-30B schematically illustrate a display system for projecting an image stream onto a user's eye. 30A-30B schematically illustrate a display system for projecting an image stream onto a user's eye.

FIG. 30C schematically illustrates a cross-sectional view of an augmented reality (AR) system.

FIG. 30D shows a simplified block diagram of a display system.

FIG. 31A schematically illustrates the operating principle of the first relay lens assembly in the display system illustrated in FIGS. 30A-30B.

31B schematically illustrates the operating principle of the second relay lens assembly in the display system illustrated in FIGS. 30A-30B.

31C-31D schematically illustrate a display system. 31C-31D schematically illustrate a display system.

32A-32C schematically illustrate a display system. 32A-32C schematically illustrate a display system. 32A-32C schematically illustrate a display system.

33A-33B schematically illustrate a display system. 33A-33B schematically illustrate a display system.

34A-34B schematically illustrate a display system. 34A-34B schematically illustrate a display system.

FIG. 35 schematically illustrates a display system.

FIG. 36A schematically illustrates an augmented reality eyepiece display system.

FIG. 36B schematically illustrates another augmented reality eyepiece display system.

FIG. 37A is a schematic diagram of a 2x afocal magnifying lens.

FIG. 37B is a schematic diagram of a bifocal magnifying afocal magnifying lens.

38A-38B schematically illustrate exemplary configurations of images that may be presented to a user.

39A-39B illustrate some example images that may be presented to a user. 39A-39B illustrate some example images that may be presented to a user.

40A-40D schematically illustrate a display system for projecting an image stream onto a user's eye. 40A-40D schematically illustrate a display system for projecting an image stream onto a user's eye. 40A-40D schematically illustrate a display system for projecting an image stream onto a user's eye. 40A-40D schematically illustrate a display system for projecting an image stream onto a user's eye.

41A-41D schematically illustrate a display system for projecting an image stream onto a user's eye. 41A-41D schematically illustrate a display system for projecting an image stream onto a user's eye. 41A-41D schematically illustrate a display system for projecting an image stream onto a user's eye. 41A-41D schematically illustrate a display system for projecting an image stream onto a user's eye.

FIG. 42 illustrates an exemplary frame structure for high FOV low resolution image streams and low FOV high resolution image streams that are time division multiplexed.

FIG. 43 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 44 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 45 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 46 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 47 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 48 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 49 schematically illustrates a display system for projecting an image stream onto a user's eye.

FIG. 50 schematically illustrates a display system for projecting an image stream onto a user's eye, according to some embodiments.

FIG. 51 schematically illustrates a display system for projecting an image stream onto a user's eye, according to some embodiments.

52A-52B schematically illustrate a display system for projecting an image stream onto a user's eye, according to some embodiments. 52A-52B schematically illustrate a display system for projecting an image stream onto a user's eye, according to some embodiments.

53A-53B schematically illustrate a display system for projecting an image stream onto a user's eye, according to some embodiments. 53A-53B schematically illustrate a display system for projecting an image stream onto a user's eye, according to some embodiments.

  Rendering virtual content for augmentation and virtual display systems is computationally intensive. Among other things, calculated strength undesirably requires the use of powerful processing units that can use large amounts of memory, can result in long latencies, and / or can have high costs and / or high energy consumption. You can

  In some embodiments, the methods and systems save computing resources such as memory and processing time by reducing the resolution of virtual content that is located away from the fixation point of the user's eye. For example, the system may render virtual content at or near the fixation point of the user's eye at a relatively high (eg, highest) resolution, while for virtual content away from the fixation point, one or more lower resolutions. To use. The virtual content is presented by a display system that may display the virtual content on multiple different depths (eg, multiple different depth planes, such as two or more depth planes), the reduction in resolution preferably being at least z. Occurring along the-axis, the z-axis is the depth axis (corresponding to the distance away from the user). In some embodiments, the resolution reduction occurs along the z-axis and one or both of the x and y axes, where the x-axis is the lateral axis and the y-axis is the vertical axis.

  Determining an appropriate resolution of the virtual content may include determining a fixation point of the user's eye in the three-dimensional space. For example, the fixation point may be the x, y, z coordinates within the user's field of view over which the user's eye is fixed. The display system may be configured to present virtual objects that have a difference in resolution, with the resolution decreasing with decreasing proximity of the virtual object and the fixation point, in other words, the resolution from the fixation point. Decreases with increasing distance.

  As discussed herein, the display system may present virtual objects within the display frustum of the display system, and virtual objects may be presented on different depth planes. In some embodiments, the display frustum is the field of view provided by the display system over which the display system is configured to present virtual content to a user of the display system. The display system may be a head-mounted display system that includes one or more waveguides capable of presenting virtual content (eg, virtual objects, graphics, text, etc.) and one or more waveguides. The tube is configured to output light with different binocular differences corresponding to different wavefront divergence and / or different depth planes (eg, corresponding to a particular distance from the user). It should be appreciated that each eye may have one or more waveguides associated with it. Using different wavefront divergence and / or different binocular divergence, the display system causes the first virtual object to appear to be located at the first depth within the user's field of view while the second virtual object is displayed by the user. It may appear to be located at a second depth within the field of view. In some embodiments, the depth plane of the fixation point or its adjacent depth plane may be determined, and the resolution of the content on other depth planes is the distance from those depth planes to the depth plane where the fixation point is located. Can be reduced based on It should be understood that references herein to virtual content depth (distance of virtual content from the user on the z-axis) refers to the apparent depth of the virtual content as it is intended to be viewed by the user. In some embodiments, the depth of a virtual object may be understood as the distance from the user of a real object that has a wavefront divergence and / or binocular dissimilarity to that of the virtual object.

  The proximity of the virtual object to the fixation point may be determined by various measurements, non-limiting examples of which include determining the distance between the fixation point and the virtual object, virtual to the resolution adjustment zone occupied by the fixation point. Determining the resolution adjustment zone occupied by the object (in an embodiment, the user's field of view is subdivided into resolution adjustment zones, as described below), and the angular proximity of the virtual object and the user's fixation point. Should be understood to include the decision of Proximity may also be determined using a combination of the above techniques. For example, the distance and / or angular proximity of the first zone (where the virtual object is located) and the second zone (where the fixation point is located) may be used to determine the proximity. These various measurements are discussed further below.

  In some embodiments, determining the fixation point may include anticipating the fixation point of the user's eye and utilizing the expected fixation point as the fixation point for determining the resolution of the virtual content. .. For example, the display system may render the particular content at a relatively high resolution, with the expectation that the user's eyes will gaze at the content. As an example, it should be appreciated that the human visual system may be sensitive to sudden changes in the scene (eg, sudden movements, changes in brightness, etc.). In some embodiments, the display system is of a type in which virtual content will cause the user's eyes to fixate it (eg, with motion in a scene where other virtual and real objects are stationary). If so, then the user's eyes may then render the virtual content in high resolution, with the expectation that it will be in focus.

  As mentioned above, in some embodiments, the determined fixation point-to-virtual object distance may correspond to a distance extending in three dimensions. As an example, the first virtual object, which is located at the same depth (for example, the same depth plane) from the determined fixation point and the user, but which is positioned horizontally or vertically from the fixation point, is also the second virtual object. Since the virtual objects of are located at a greater depth (eg, a farther depth plane) from the determined fixation point, they may be similarly reduced in resolution. As a result, different resolutions may be associated with different distances from the fixation point.

  In some embodiments, the environment around the user may be divided into a volume of space (also referred to herein as a resolution adjustment zone), and the resolution of virtual objects within the same resolution adjustment zone is Similar. The resolution adjustment zone may have an arbitrary three-dimensional shape, such as a cube, or other three-dimensional polygonal shape, or a curved three-dimensional shape, as described herein. In some embodiments, all resolution adjustment zones have a similar shape, eg, a cuboid or a sphere. In some other embodiments, different resolution adjustment zones may have different shapes or sizes (eg, the shape and / or size of the volume may change with distance from the fixation point).

  In some embodiments, the resolution adjustment zone is part of the user's field of view. For example, the user's field of view may be separated into volumes of space that form a resolution adjustment zone. In some embodiments, each depth plane may be subdivided into one or more continuous spatial volumes, ie, one or more resolution adjustment zones. In some embodiments, each resolution adjustment zone is a specific range of depth from the user (eg, a depth plane value +/− some variance, examples of variance being 0.66 dpt, 0.50 dpt, (Including 0.33 dpt, or 0.25 dpt) and certain lateral and certain vertical distances. A virtual object located in the same resolution adjustment zone as the determined fixation point can be presented (eg, rendered) with high (eg, full) resolution, while being located in a volume of space outside the fixation adjustment zone. The virtual object may be rendered at a lower resolution according to the distance of the volume from the volume of the space of the fixation point. In some embodiments, each resolution adjustment zone may be assigned a particular resolution (eg, a particular reduction in resolution over full resolution), and virtual content falling within a given zone is associated with that zone. Can be rendered with a given resolution. In some embodiments, the distance between a volume and the volume occupied by the fixation point can be determined and the resolution can be set based on this distance.

  Advantageously, the number and size of resolution adjustment zones utilized to segment the user's field of view can be modified according to the confidence in the user's determined fixation point. For example, the size associated with the volume of each space may be increased or decreased based on the confidence that the user's line of sight is converging on a precise point in three-dimensional space. If the reliability at the fixation point is high, the display system may present only virtual objects within a certain compact resolution adjustment zone at a relatively high resolution (compact resolution adjustment zone including the fixation point) while other virtual objects. , And thus save processing power. However, if the confidence is low, the display system may increase the size of the volume of each space so that the volume of each space contains more virtual objects within the volume of the fixation space (eg, Reduce the overall number of volumes). The size and shape of the volume may be fixed during the production of the display system, for example based on the tolerances expected in the system for determining the fixation point, and / or the characteristics of the user, the environment of the user, It should be appreciated that it may be adjusted or set in the field in response to changes in software that change the tolerances for the system for determining the fixation point and / or fixation.

  It should be appreciated that the user's sensitivity to resolution may decrease with distance from the fixation point. As a result, the perceptual ability of the reduction in resolution is reduced or reduced by ensuring that full resolution content is presented to the fixation point, and by allowing for error tolerances as to where the fixation point is located. It may be eliminated, thereby providing the perception of high resolution displays without utilizing the computational resources typically required to present content for such high resolution displays.

  In some embodiments, the proximity of the virtual object and the fixation point may be determined based on the angular proximity of the virtual object and the user's line of sight, and the resolution of the virtual object decreases as the angular proximity decreases. You can In some embodiments, this may result in virtual objects located at different depths being presented to the user at similar resolutions. For example, the first virtual object at a location corresponding to the user's determined fixation point may be located in front of the second virtual object (eg, the depth is closer to the user). The second virtual object is along the line of sight of the user, and thus similarly the user's eye will be on the fovea of the user most sensitive to the change in resolution, so the second virtual object is , May optionally be presented at a resolution similar (eg, the same) as the first virtual object. Optionally, the second virtual object is reduced in resolution and may be further adjusted via a blurring process, which may represent that the second virtual object is further from the user (eg, located on a farther depth plane). (Eg, a Gaussian blur kernel may be convolved with the second virtual object).

  The reduction in resolution may vary based on how the virtual content is presented by the display system. In some embodiments, a first exemplary display system, referred to herein as a variable focus display system, may present virtual content on different depth planes and all content (eg, virtual objects). Are presented in the same depth plane at one time, for example, every frame presented to the user (eg, via the same waveguide). That is, a variable focus display system may have a single depth plane (eg, selected from multiple depth planes based on the user's fixation point, or based on the depth of a particular presented virtual object). It may be used at one time to present content, and the depth plane may change in subsequent frames (eg, select a different depth plane). In some other embodiments, a second exemplary display system, referred to herein as a multifocal display system, may present virtual content on different depth planes, the content being simultaneously Displayed on multiple depth planes. As will be further described herein, the variable focus display system may optionally utilize a single frame buffer, with respect to the above example of blurring the second virtual object The two virtual objects may be blurred prior to presentation to the user from the single frame buffer. In contrast, a multifocal display system may also present a second virtual object on a depth farther from the first virtual object (eg, on a depth plane further away), optionally at a reduced resolution. Often, the second virtual object may appear to the user as being blurred (eg, the second virtual object may be blurred based on the natural physics of the user's eye without further processing. Ah).

  As disclosed herein, a display system may present virtual objects at or near a determined fixation point with relatively high (eg, full) resolution, and virtual objects further from the fixation point. May be presented with reduced resolution. Preferably, the relatively high resolution is the highest resolution for the presentation of virtual objects within the user's field of view. The relatively high resolution may be the maximum resolution of the display system, a user-selectable resolution, a resolution based on the specific computing hardware presenting the virtual object, etc.

  It should be appreciated that adjusting the resolution of the virtual object may include any modification to the virtual object to alter the presentation quality of the virtual object. Such modifications include adjusting the polygon count of the virtual object, including adjusting the quality at one or more points in the graphics pipeline of the graphics processing unit (GPU), utilized to generate the virtual object. Adjusting the primitives that are performed (eg, adjusting the shape of the primitives, eg, adjusting the primitives from a triangular mesh to a quadrilateral mesh, etc.), adjusting the operations performed on the virtual object (eg, Shader operation), adjusting texture information, adjusting color resolution or depth, adjusting the number of rendering cycles or frame rate, and the like.

  In some embodiments, on the x and y-axes, changes in resolution of virtual content away from the fixation point may generally track changes in distribution of photoreceptors within the retina of the user's eye. For example, it should be appreciated that views of the world and virtual content may be imaged on the retina such that different parts of the retina may be mapped to different parts of the user's field of view. Advantageously, the resolution of virtual content across the user's field of view may generally track the density of corresponding photoreceptors (rods or cones) across the retina. In some embodiments, the resolution reduction away from the fixation point may generally track the reduction in cone density across the retina. In some other embodiments, resolution reduction away from fixation may generally track reductions in rod density across the retina. In some embodiments, the tendency for resolution reduction away from the fixation point is ± 50%, ± 30%, ± 20%, or ± of the tendency for reduction in rod and / or cone density across the retina. It can be within 10%.

  Rods and cones are active at different levels of incidence. For example, cones are active under relatively bright conditions, while rods are active under relatively low light conditions. As a result, the reduction in resolution generally tracks the density of rods or cones across the retina, and in some embodiments, the display system is configured to determine the amount of light incident on the retina. May be done. Based on the amount of this light, an appropriate adjustment in resolution can be made. For example, reductions in resolution generally track changes in rod density across the retina under low light conditions, while reductions in resolution generally track changes in cone density under bright conditions. obtain. As a result, in some embodiments, the display system may be configured to change the profile of reduction in image resolution based on the amount of light incident on the retina.

  It is to be appreciated that the human eye's ability to resolve subtle details may not be directly proportional to the density of rods or cones within the retina. In some embodiments, changes in resolution of virtual content across the user's field of view generally track changes in the ability of the eye to resolve fine details. As mentioned above, the rate of change of resolution of virtual content may vary with the amount of light reaching the retina.

  In some embodiments, the amount of light reaching the retina may be determined by detecting the amount of ambient light incident on a sensor mounted on the display device. In some embodiments, determining the amount of light reaching the retina may also include determining the amount of light output by the display device to the user. In yet another embodiment, the amount of light reaching the retina may be determined by imaging the user's eyes and determining the pupil size. Since pupil size is related to the amount of light reaching the retina, determining pupil size allows the amount of light reaching the retina to be extrapolated.

  It should be appreciated that full-color virtual content may be formed by a plurality of primary color images that together provide a full-color perception. The human eye can have different sensitivities to different wavelengths or colors of light. In some embodiments, in addition to changing based on proximity to the fixation point, the change in resolution of the virtual content may change based on the color of the primary color image presented by the display system. For example, if the primary color images comprise red, green, and blue images, the green primary color image may have higher resolution than the red primary color image, which may have higher resolution than the blue primary color image. In some embodiments, the amount of light reaching the retina may be determined to account for changes in the eye's sensitivity to different colors at different levels of incident light, for a given primary color image. Resolution adjustments can also vary based on determining the amount of light reaching the retina.

  It should be appreciated that the contrast sensitivity of the eye may also vary based on the amount of light incident on the retina. In some embodiments, the size or total number of tones in the contrast within the virtual content may vary based on the amount of light reaching the retina. In some embodiments, the contrast ratio of the image forming the virtual content may vary based on the amount of light incident on the retina, the contrast ratio decreasing with decreasing amount of light.

  In some embodiments, some portion of the user's field of view may not be provided with any virtual content. For example, the display system may be configured to not provide virtual content within the blind spots caused by the optic nerve and / or peripheral blind spots of a given eye.

  As discussed herein, a display system may be configured to display high resolution content in one portion of the user's field of view and lower resolution content in another portion of the user's field of view. It should be appreciated that high resolution content may have a higher pixel density than lower resolution content. In some circumstances, the display system may be configured to provide such high and low resolution content by effectively superimposing high resolution and low resolution images. For example, the system may display a low resolution image that spans the entire field of view and then displays a high resolution image that spans a small portion of the field of view, where the high resolution image is co-located with the corresponding portion of the low resolution image. To position. High and low resolution images can be routed through different optics that output light at appropriate angles and determine the amount of field of view they occupy.

  In some embodiments, a single spatial light modulator (SLM) may be used to encode the light with image information and a beam splitter or optical switch may be used to split the single light stream from the SLM into two. It may be used to split into streams, one stream propagating through the optics for the low resolution image and a second stream propagating through the optics for the high resolution image. In some other embodiments, the polarization of the light encoded with the image information is selectively switched, effectively providing different angular magnification factors for light of different polarizations, thereby increasing high and low. It may be passed through an optical system that provides a resolution image.

  Advantageously, the various embodiments disclosed herein reduce the processing power requirements for providing content on a display system. A larger allocation of processing power may be spent on virtual objects closer to the user's 3D fixation point, while processing power for distant virtual objects may be reduced, thus reducing the overall demand for the display system. Processing power is reduced, and thus the size of the processing component, the heat generated by the processing component, and the energy requirements for the display system (eg, the display system is optionally battery powered and a lower capacity battery). , And / or may operate for longer durations for a given battery). Thus, the embodiments described herein address the technical issues arising from augmented or virtual reality display systems. In addition, the described techniques ensure that the graphical content is presented differently (eg, the resolution is modified) in response to the presentation to the user, while the graphical content is the same. Manipulate the graphical content as it appears to the user. Thus, the display system preserves visual fidelity and saves processing power while translating graphical content as the user overlooks its surroundings.

  It should be appreciated that the display system may be part of an augmented reality display system or a virtual reality display system. As an example, the display of the display system may be transparent, allowing the user to see the real world while providing virtual content to the user in the form of images, videos, interactions, etc. You may. As another example, the display system may block the user's real-world view and virtual reality images, videos, interactions, etc. may be presented to the user.

  Referring now to the drawings, like reference numbers refer to like parts throughout.

  FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. The user's eyes are spaced and when looking at a real object in space, each eye has a slightly different view of the object and may form an image of the object at different locations on the retina of each eye. Please understand that. This may be referred to as binocular parallax and may be utilized by the human visual system to provide depth perception. Conventional display systems provide slightly different views of the same virtual object for each eye 210, 220, corresponding to the view of the virtual object that each virtual eye would see as if it were a real object at the desired depth. Binocular disparity is simulated by presenting two distinctly different images 190, 200 with it. These images provide binocular cues that the user's visual system can interpret to derive the perception of depth.

  With continued reference to FIG. 2, the images 190, 200 are separated from the eyes 210, 220 by a distance 230 on the z-axis. The z-axis is parallel to the optical axis of the viewer with his eyes gazing at the object at optical infinity immediately in front of him. The images 190, 200 are flat and at a fixed distance from the eyes 210, 220. Based on the slightly different views of the virtual object in the images presented to the eyes 210, 220, respectively, the eye will inevitably come to a single point when the image of the object comes to the corresponding point on each retina of the eye. It can rotate to maintain binocular vision. This rotation may focus the line of sight of each eye 210, 220 onto a point in space where the virtual object is perceived as being present. As a result, the provision of three-dimensional images conventionally provides binocular cues that can manipulate the vergence and divergence movements of the user's eyes 210, 220, which the human visual system interprets to provide depth perception. With that.

  However, generating a realistic and comfortable perception of depth is difficult. It should be appreciated that light from an object at different distances from the eye will have wavefronts with different amounts of divergence. 3A-3C illustrate the relationship between distance and ray divergence. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in FIGS. 3A-3C, the rays become more divergent as the distance to the object decreases. Conversely, as the distance increases, the rays become more collimated. In other words, the light field produced by a point (object or part of an object) can be said to have a spherical wavefront curvature, which is a function of the distance the point is away from the user's eye. Although only monocular 210 is illustrated in FIGS. 3A-3C and various other figures herein for clarity of illustration, the discussion regarding eye 210 applies to both eyes 210 and 220 of the viewer. obtain.

  With continued reference to FIGS. 3A-3C, light from an object that the viewer's eye is gazing at may have different wavefront divergence. Due to the different wavefront divergence, light can be focused differently by the crystalline lens of the eye, which in turn causes the crystalline lens to assume different shapes and form a focused image on the retina of the eye. You can request. If the focused image is not formed on the retina, the resulting retinal blurring is a cue for accommodation that causes a change in the shape of the lens of the eye until a focused image is formed on the retina. Acts as. For example, accommodation cues trigger relaxation or contraction of the ciliary muscles that surround the lens of the eye, thereby modulating the force applied to the low ligament that holds the lens, and thus the fixation. Changing the shape of the lens of the eye until the retinal blurring of the image being viewed is eliminated or minimized, thereby providing a focused image of the object being fixed on the retina of the eye (eg, the fovea). Can be formed into. The process by which the lens of the eye changes shape can be referred to as accommodation and is required to form a focused image of the object being fixed on the retina (eg, fovea) of the eye. The shape of the lens may be referred to as accommodation.

  Referring now to FIG. 4A, a representation of the accommodation-convergence-divergence motor response of the human visual system is illustrated. Movement of the eye to fixate the object causes the eye to receive light from the object, which forms an image on each of the retinas of the eye. The presence of retinal blur in the image formed on the retina may provide cues for accommodation and the relative location of the image on the retina may provide cues for vergence and divergence movements. The cues for accommodating produce accommodative effects that result in a particular accommodative condition in which the lens of the eye forms a focused image of the object on the retina (eg, fovea) of the eye. On the other hand, the cue for vergence / divergence movement is such that the image formed on each retina of each eye is located at the corresponding retinal point that maintains a single binocular vision. Rotation). At these positions, it can be said that the eye is in a specific vergence / divergence movement state. With continued reference to FIG. 4A, accommodation can be understood as the process by which the eye achieves a particular accommodation state, and vergence-divergence movement is where the eye achieves a particular convergence-divergence movement state. Can be understood as a process of doing. As shown in FIG. 4A, the accommodation and vergence / divergence movement states of the eye may change when the user gazes at another object. For example, the accommodation state may change if the user is gazing at a new object at a different depth on the z-axis.

  Without being limited by theory, it is believed that the viewer of the object may perceive the object as "three-dimensional" due to the combination of vergence / divergence movement and accommodation. As described above, the convergence / divergence movement of the two eyes relative to each other (for example, the rotation of the eyes such that the pupils move toward or away from each other to converge the eye's line of sight and fix the object). ) Is closely associated with accommodation of the lens of the eye. Under normal conditions, changing the shape of the lens of the eye and changing the focus from one object to another at different distances is automatically known as "accommodation-convergence / divergent motion reflex". In relation, it will cause consistent changes in vergence / divergence movements to the same distance. Similarly, changes in vergence-divergence movements will trigger consistent changes in lens shape under normal conditions.

  Referring now to FIG. 4B, an example of different accommodation and vergence-divergence movement states of the eye is illustrated. The pair of eyes 222a gazes at the object at optical infinity, while the pair of eyes 222b gazes at the object 221 below optical infinity. It should be noted that the vergence / divergence movement states of each pair of eyes are different, and the pair of eyes 222a are directed straight, while the pair of eyes 222 converge on the object 221. The accommodation states of the eyes forming each pair of eyes 222a and 222b are also different, as represented by the different shapes of lenses 210a, 220a.

  Undesirably, many users of conventional "3-D" display systems find such conventional systems due to the mismatch between accommodation and vergence / divergence motion states in these displays. May be found uncomfortable or may not perceive a sense of depth at all. As mentioned above, many stereoscopic or "3-D" display systems display a scene by providing each eye with a slightly different image. Such systems provide, among other things, merely different presentations of the scene, causing changes in the vergence and divergence movement states of the eye, but without corresponding changes in their accommodation state. Therefore, it is uncomfortable for many viewers. Rather, the image is presented at a fixed distance from the eye by the display so that the eye sees all image information in a single accommodation state. Such an arrangement defeats the "accommodation-convergence-divergence reflex" by causing a change in vergence-divergence-motion state without a matching change in accommodation. This inconsistency is considered to cause viewer discomfort. Providing a better match between accommodation and vergence-divergence movement, a display system can form a more realistic and comfortable simulation of a three-dimensional image.

  Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes and providing depth perception. As a result, a highly plausible simulation of the perceived depth can be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. In some embodiments, different presentations provide both cues for vergence and divergence movements and matching cues for accommodation, thereby providing physiologically correct accommodation-convergence and divergence movements. Matching may be provided.

  With continued reference to FIG. 4B, two depth planes 240 are illustrated, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, vergence and divergence movement cues may be provided by displaying images from different viewpoints for each eye 210, 220 as appropriate. In addition, for a given depth plane 240, the light forming the image provided to each eye 210, 220 has a wavefront divergence corresponding to the light field produced by a point at that depth plane 240 distance. May be.

  In the illustrated embodiment, the distance along the z-axis of the depth plane 240 that contains the point 221 is 1 m. As used herein, distance or depth along the z-axis may be measured with a zero point located at the exit pupil of the user's eye. Therefore, the depth plane 240 located at a depth of 1 m corresponds to a distance 1 m away from the exit pupil of the user's eye on the optical axis of those eyes. As an approximation, the depth or distance along the z-axis is measured from the display in front of the user's eye (eg, the surface of the waveguide), and with the device and eye oriented toward optical infinity. A value for the distance to the exit pupil of the user's eye may be added. That value is called the pupil distance and may correspond to the distance between the exit pupil of the user's eye and the display worn by the user in front of the eye. In practice, the value for pupil distance may generally be a normalized value used for all viewers. For example, the pupil distance may be assumed to be 20 mm and the depth plane at a depth of 1 m may be at a distance of 980 mm in front of the display.

  4C and 4D, examples of matched accommodation-convergence-divergence kinematics and mismatched accommodation-convergence-divergence kinematics are illustrated, respectively. As shown in FIG. 4C, the display system may provide an image of the virtual object to each eye 210, 220. The image may cause the eyes 210, 220 to assume a vergence-divergence motion state in which the eyes converge on the point 15 on the depth plane 240. In addition, the image may be formed by light having a wavefront curvature corresponding to the real object in its depth plane 240. As a result, the eyes 210, 220 are in an accommodative state with the image focused on the retinas of their eyes. Thus, the user may perceive the virtual object as being at point 15 on the depth plane 240.

It should be appreciated that each of the accommodation and vergence / divergence movement states of the eye 210, 220 is associated with a particular distance on the z-axis. For example, an object at a particular distance from the eyes 210, 220 will cause those eyes to assume a particular accommodation state based on the distance of the object. The distance associated with a particular accommodation state may be referred to as the accommodation distance A _d . Similarly, there are also specific vergence-divergence movement distances V _d associated with the eyes, in particular vergence-divergence movement states or positions with respect to each other. If the accommodation distance and vergence / divergence movement distance are matched, the relationship between accommodation and vergence / divergence movement may be said to be physiologically correct. This is considered the most comfortable scenario for the viewer.

However, in a stereoscopic display, the accommodation distance and the convergence / divergence movement distance may not always match. For example, as illustrated in FIG. 4D, the image displayed on the eye 210, 220 may be displayed with a wavefront divergence corresponding to the depth plane 240, which eye 210, 220 may point 15a on the depth plane. , 15b may be in focus, a particular accommodation state. However, the images displayed on the eyes 210, 220 may provide cues for vergence and divergence movements that cause the eyes 210, 220 to converge on a point 15 that is not located on the depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to the distance from the user's particular reference point (eg, the exit pupil of the eye 210, 220) to the depth plane 240 while the convergence / divergence movement distance. Corresponds to a larger distance from that reference point to point 15. Therefore, the accommodation distance differs from the convergence / divergence movement distance, and there is a accommodation-convergence / divergence movement mismatch. Such inconsistencies are considered undesirable and can cause discomfort to the user. It should be appreciated that the mismatch corresponds to the distance (eg, V _d −A _d ) and can be characterized using diopters (units of reciprocal length 1 / m). For example, 1.75 _{V d} and 1.25 diopters of _{A d} or 1.25 _{V d} and 1.75 diopters of _{A d} diopter diopters is 0.5 accommodation diopter - Congestion & divergence movement mismatch Will provide.

  In some embodiments, reference points other than the exit pupil of the eyes 210, 220 may have accommodation-convergence / divergence movements, as long as the same reference points are utilized for accommodation and convergence / divergence movement distances. It should be appreciated that it may be utilized to determine the distance to determine the mismatch. For example, the distance may be measured from the cornea to the depth plane, the retina to the depth plane, the eyepiece (eg, the waveguide of the display device) to the depth plane, and so on.

  Without being limited by theory, the user may still find that the inconsistency itself does not cause significant discomfort, and still up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters. It is considered that the accommodation-convergence / divergence movement misalignment can be perceived as physiologically correct. In some embodiments, a display system disclosed herein (e.g., display system 250, FIG. 6) provides an image with accommodation-convergence / divergence motion misalignment of about 0.5 diopters or less. To the viewer. In some other embodiments, the accommodation-convergence / divergence movement mismatch of the image provided by the display system is about 0.33 diopters or less. In yet another embodiment, the accommodation-convergence / divergence motion mismatch of the image provided by the display system is about 0.25 diopters or less and includes about 0.1 diopters or less.

  FIG. 5 illustrates aspects of an approach for simulating a three-dimensional image by modifying the wavefront divergence. The display system includes a waveguide 270 that is configured to receive light 770 encoded with image information and output the light to the user's eye 210. The waveguide 270 may output light 650 with a defined amount of wavefront divergence corresponding to the wavefront divergence of the light field produced by a point on the desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on that depth plane. In addition, the other eye of the user will be illustrated so that it can be provided with image information from a similar waveguide.

  In some embodiments, a single waveguide may be configured to output light with a set wavefront divergence corresponding to a single or limited number of depth planes, and / or waveguides. The tube may be configured to output light in a limited range of wavelengths. As a result, in some embodiments, multiple or stack waveguides are utilized to provide different wavefront divergence for different depth planes and / or to output different ranges of wavelengths of light. Good. It should be appreciated that as used herein, the depth plane may follow the contours of flat or curved surfaces. In some embodiments, for convenience, the depth plane may follow the contours of a flat surface.

  FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. The display system 250 uses a plurality of waveguides 270, 280, 290, 300, 310 to provide a stack of waveguides or stacked guides that can be utilized to provide three-dimensional perception to the eye / brain. A wave tube assembly 260 is included. It should be appreciated that the display system 250 may be considered a light field display in some embodiments. In addition, the waveguide assembly 260 may also be referred to as an eyepiece.

  In some embodiments, the display system 250 may be configured to provide a substantially continuous cue for vergence and divergence movements and a plurality of discrete cues for accommodation. Cues for vergence and divergence movements may be provided by displaying different images on each of the user's eyes, and cues for accommodation form images with selectable discrete amounts of wavefront divergence. May be provided by outputting light that does. In other words, the display system 250 may be configured to output light with a variable level of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a particular depth plane and may be provided by a particular one of the waveguides 270, 280, 290, 300, 310.

  With continued reference to FIG. 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 are configured to transmit image information to the eye with varying levels of wavefront curvature or ray divergence. May be. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image injecting devices 360, 370, 380, 390, 400 may function as a light source for the waveguide, and in order to inject image information into the waveguide 270, 280, 290, 300, 310. Each may be utilized and may be configured to disperse incident light across each individual waveguide for output towards the eye 210, as described herein. Light exits the output surfaces 410, 420, 430, 440, 450 of the image input devices 360, 370, 380, 390, 400 and the corresponding input surfaces 460 of the waveguides 270, 280, 290, 300, 310. 470, 480, 490, 500. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be the edge of the corresponding waveguide, or a portion of the corresponding major surface of the waveguide (ie, the world). 510 or one of the waveguide surfaces directly facing the viewer's eye 210). In some embodiments, a single beam of light (eg, a collimated beam) is launched into each waveguide and corresponds to a particular angle (( And the divergence) may be output toward the eye 210 and output the entire field of the cloned collimated beam. In some embodiments, a single one of the image injecting devices 360, 370, 380, 390, 400 may include multiple (eg, three) waveguides 270, 280, 290, 300, 310. It may be associated with and throw light into it.

  In some embodiments, the image injection devices 360, 370, 380, 390, 400 each generate image information for injection into a corresponding waveguide 270, 280, 290, 300, 310, respectively. , A discrete display. In some other embodiments, the image input devices 360, 370, 380, 390, 400 may transfer the image information, eg, via one or more optical conduits (such as fiber optic cables), to the image input devices 360, 370. , 380, 390, 400, respectively, at the output of a single multiplexed display. It is understood that the image information provided by the image injection devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (eg, different primary colors, as discussed herein). I want to.

  In some embodiments, the light launched into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520, which comprises a light module 530, which comprises a light emitting diode. It may include a light emitter such as an (LED). Light from the light module 530 may be directed and modified by the light modulator 540, eg, a spatial light modulator, via the beam splitter 550. The light modulator 540 may be configured to change the perceived intensity of light injected into the waveguides 270, 280, 290, 300, 310 and encode the light with image information. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays. Image injection devices 360, 370, 380, 390, 400 are schematically illustrated and, in some embodiments, these image injection devices associate light with waveguides 270, 280, 290, 300, 310. It is to be understood that different light paths and locations within the common projection system that are configured to output into the same can be represented. In some embodiments, the waveguide of the waveguide assembly 260 can act as an ideal lens while relaying the light injected into the waveguide to the user's eye. In the present concept, the object may be the spatial light modulator 540 and the image may be an image on the depth plane.

  In some embodiments, the display system 250 directs light in various patterns (eg, raster scan, spiral scan, Lissajous pattern, etc.) into one or more waveguides 270, 280, 290, 300, 310. Finally, it may be a scanning fiber display comprising one or more scanning fibers configured to project into the viewer's eye 210. In some embodiments, the illustrated image injection device 360, 370, 380, 390, 400 is adapted to inject light into one or more waveguides 270, 280, 290, 300, 310. The configured single scan fiber or bundle of scan fibers may be represented diagrammatically. In some other embodiments, the illustrated image injection device 360, 370, 380, 390, 400 may schematically represent a plurality of scanning fibers or a bundle of scanning fibers, each guiding light. Configured for loading into an associated one of tubes 270, 280, 290, 300, 310. It should be appreciated that the one or more optical fibers may be configured to transmit light from the optical module 530 to the one or more waveguides 270, 280, 290, 300, 310. One or more intervening optical structures are provided between the scanning fiber or fibers and the one or more waveguides 270, 280, 290, 300, 310 to, for example, direct light exiting the scanning fiber. It should be appreciated that one or more waveguides 270, 280, 290, 300, 310 may be redirected.

  The controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image injection devices 360, 370, 380, 390, 400, the light source 530, and the light module 540. In some embodiments, controller 560 is part of local data processing module 140. The controller 560 adjusts the timing and provision of image information to the waveguides 270, 280, 290, 300, 310, eg, according to any of the various schemes disclosed herein, programming (eg, non-compliant). Instructions in a transitory medium). In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. Controller 560 may be part of processing module 140 or 150 (FIG. 9D) in some embodiments.

  With continued reference to FIG. 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Each of the waveguides 270, 280, 290, 300, 310 is planar or otherwise shaped with major top and bottom surfaces and edges extending between those major top and bottom surfaces ( For example, it may have a curvature. In the illustrated configuration, each of the waveguides 270, 280, 290, 300, 310 redirects light, propagates within each individual waveguide, and outputs image information from the waveguides to the eye 210. May include outcoupling optical elements 570, 580, 590, 600, 610 configured to extract light from the waveguide. The extracted light may also be referred to as outcoupling light and the outcoupling optical element may also be referred to as light extraction optical element. The extracted beam of light may be output by the waveguide at a location where the light propagating in the waveguide strikes the light extraction optical element. Outcoupling optical elements 570, 580, 590, 600, 610 may be, for example, gratings that include diffractive optical features as discussed further herein. For ease of illustration and clarity of illustration, the waveguide 270, 280, 290, 300, 310 is shown positioned on the bottom major surface of the waveguide, although in some embodiments the outcoupling optical element 570 is shown. , 580, 590, 600, 610 may be located on the top and / or bottom major surfaces, and / or waveguides 270, 280, 290, 300, 310, as discussed further herein. May be placed directly in the volume of the. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are formed in layers of material that are attached to a transparent substrate and form the waveguides 270, 280, 290, 300, 310. May be done. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be monolithic pieces of material and the outcoupling optical elements 570, 580, 590, 600, 610 may be made of that material. It may be formed on and / or inside the component.

  With continued reference to FIG. 6, as discussed herein, each waveguide 270, 280, 290, 300, 310 outputs light to form an image corresponding to a particular depth plane. Is configured as follows. For example, the waveguide 270 proximate to the eye may be configured to deliver collimated light (cast into such a waveguide 270) to the eye 210. The collimated light may represent the optical infinity focal plane. The next upper waveguide 280 may be configured to deliver collimated light that passes through the first lens 350 (eg, a negative lens) before it can reach the eye 210. Such a first lens 350 is interpreted such that the eye / brain then emits the light emanating from the upper waveguide 280 from a first focal plane closer inward from optical infinity toward the eye 210. To generate some convex wavefront curvature. Similarly, the third upper waveguide 290 allows its output light to pass through both the first 350 and second 340 lenses before reaching the eye 210. The combined optical power of the first 350 and second 340 lenses is such that the eye / brain is optical infinity where the light emanating from the third waveguide 290 was the light from the next upper waveguide 280. It may be configured to generate another incremental amount of wavefront curvature to be interpreted as originating from a second focal plane that is closer inward toward the person from a distance.

  The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack having its output visible to the eye due to the aggregate focal power, which represents the focal plane closest to the person. Deliver through all of the lenses between and. A compensating lens layer 620 is provided on top of the stack to compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light originating from the other world 510 of the stacked waveguide assembly 260. It may be arranged to compensate for the collective power of the lower lens stack 320, 330, 340, 350. Such a configuration provides as many perceived focal planes as waveguide / lens pairs available. Both the outcoupling optical element of the waveguide and the focusing side of the lens may be static (ie, not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

  In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set in the same depth plane, or waveguides 270, 280, 290, 300, 310. May be configured to output images set in the same depth plane, with one set per depth plane. This may provide the advantage of forming tiled images to provide an extended field of view in their depth plane.

      With continued reference to FIG. 6, the outcoupling optical elements 570, 580, 590, 600, 610 redirect light from their individual waveguides due to the particular depth planes associated with the waveguides. In addition, the main light may be output with an appropriate amount of divergence or collimation. As a result, waveguides with different associated depth planes may have different configurations of outcoupling optical elements 570, 580, 590, 600, 610, which, depending on the associated depth planes. , Outputs light with different amounts of divergence. In some embodiments, the light extraction optics 570, 580, 590, 600, 610 may be volume or surface features, which may be configured to output light at a specific angle. . For example, the light extraction optics 570, 580, 590, 600, 610 may be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 may not be lenses. Rather, they may simply be spacers (eg, structures for forming cladding layers and / or voids).

  In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. ). Preferably, the DOE is sufficient such that only a portion of the light in the beam is deflected towards the eye 210 using each DOE intersection, while the rest continue to travel through the waveguide through the TIR. Has low diffraction efficiency. The light carrying the image information is thus split into several related exit beams that exit the waveguide at various locations, so that for this particular collimated beam bouncing in the waveguide There is a very uniform pattern of emission and emission towards 210.

  In some embodiments, one or more DOEs may be switchable between an actively diffracting “on” state and a significantly non-diffracting “off” state. For example, the switchable DOE may comprise a layer of polymer dispersed liquid crystal, in which the microdroplets comprise a diffraction pattern in the host medium, the refractive index of the microdroplets being substantially that of the host material. May be switched to match (in which case the pattern does not significantly diffract incident light) or the microdroplets may be switched to a refractive index that does not match that of the host medium (in which case). , The pattern actively diffracts the incident light).

  In some embodiments, a camera assembly 630 (eg, a digital camera, including visible and infrared light cameras) captures an image of the eye 210 and / or tissue surrounding the eye 210 and detects, for example, user input. And / or may be provided to monitor the physiological condition of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 also includes an image capture device and a light source that projects light (eg, infrared light) onto the eye, which can then be reflected by the eye and detected by the image capture device. Good. In some embodiments, the camera assembly 630 may be mounted on the frame 80 (FIG. 9D) and is also in electrical communication with a processing module 140 and / or 150 that may process image information from the camera assembly 630. Good. In some embodiments, one camera assembly 630 may be utilized for each eye and monitor each eye separately.

  Referring now to FIG. 7, an example of an exit beam output by a waveguide is shown. Although one waveguide is shown, other waveguides within the waveguide assembly 260 (FIG. 6) may work as well, with the waveguide assembly 260 including multiple waveguides. I want you to understand. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 strikes the DOE 570, some of the light exits the waveguide as an exit beam 650. The exit beam 650 is shown as being substantially parallel, but at an angle (eg, divergent exit beam formation), as discussed herein, and depending on the depth plane associated with the waveguide 270. It may be redirected to propagate to the eye 210. The substantially collimated output beam is a waveguide with outcoupling optics that outcouples the light to form an image that appears to be set in the depth plane at a far distance from the eye 210 (eg, optical infinity). It should be understood that can be shown. Other waveguides or sets of other outcoupling optics may output a more divergent, exiting beam pattern that causes the eye 210 to adjust to a closer distance and focus on the retina. And will be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

  In some embodiments, a full color image may be formed in each depth plane by overlaying the image on a primary color, eg, each of three or more primary colors. FIG. 8 illustrates an example of stacked waveguide assemblies, each depth plane containing an image formed using a plurality of different primary colors. The illustrated embodiment shows depth planes 240a-240f, but more or less depth is also contemplated. Each depth plane includes three or more primary colors associated with it, including a first image of a first color G, a second image of a second color R, and a third image of a third color B. It may have an image. Different depth planes are illustrated by different numbers for diopters (dpt) following the letters G, R, and B. By way of example only, the number following each of these letters indicates the diopter (1 / m), ie the inverse distance of the depth plane from the viewer, and each box in the figure represents an individual primary color image. In some embodiments, the exact location of the depth plane for different primaries may vary to account for differences in the focusing of different wavelengths of light by the eye. For example, different primary color images for a given depth plane may be placed on the depth plane corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort, and / or reduce chromatic aberration.

  In some embodiments, each primary color light may be output by a single dedicated waveguide, so that each depth plane may have multiple waveguides associated with it. In such an embodiment, each box in the figure, including the letters G, R, or B, may be understood to represent an individual waveguide, three waveguides being provided per depth plane. Alternatively, three primary color images are provided per depth plane. Although the waveguides associated with each depth plane are shown adjacent to each other in this figure for ease of illustration, in a physical device all waveguides are one waveguide per level. It should be appreciated that they may be arranged in a stack with tubes. In some other embodiments, multiple primaries may be output by the same waveguide, eg, only a single waveguide may be provided per depth plane.

  With continued reference to FIG. 8, in some embodiments G is green, R is red and B is blue. In some other embodiments, other colors associated with other wavelengths of light, including magenta and cyan, are also used in addition to one or more of red, green, or blue. Or may replace them.

  References to a given color of light throughout this disclosure are understood to include light of one or more wavelengths within the range of wavelengths of light perceived by a viewer as that given color. I want you to understand. For example, red light may include one or more wavelengths of light in the range of about 620-780 nm, and green light may include one or more wavelengths of light in the range of about 492-577 nm. Well, blue light may include light at one or more wavelengths that are in the range of about 435-493 nm.

  In some embodiments, the light source 530 (FIG. 6) may be configured to emit light at one or more wavelengths outside the visual perception range of the viewer, eg, infrared and / or ultraviolet wavelengths. . In addition, internal coupling, external coupling, and other light redirecting structures of the waveguides of display 250 direct this light from the display to the user's eye 210, for example, for imaging and / or user stimulation applications. May be configured to direct and emit.

  Referring now to FIG. 9A, in some embodiments, light impinging on the waveguide may need to be redirected in order to internally couple the light into the waveguide. In-coupling optical elements may be used to redirect and in-couple light into its corresponding waveguide. FIG. 9A illustrates a cross-sectional side view of an embodiment of a plurality or sets 660 of stacked waveguides, each including an incoupling optical element. The waveguides may each be configured to output light at one or more different wavelengths or one or more different wavelength ranges. The stack 660 may correspond to the stack 260 (FIG. 6), and the illustrated waveguides of the stack 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310. Good, but the light from one or more of the image injection devices 360, 370, 380, 390, 400 is in the waveguide from a position that requires that the light be redirected for internal coupling. Please understand that it will be put into.

  The illustrated set 660 of stacked waveguides includes waveguides 670, 680, and 690. Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input area on the waveguide), for example, the incoupling optical element 700 includes a major surface (eg, upper side) of the waveguide 670. Inner coupling optical element 710 is disposed on a major surface of waveguide 680 and inner coupling optical element 720 is disposed on a major surface of waveguide 680 (eg, upper major surface). , The upper major surface). In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be located on the bottom major surface of the individual waveguides 670, 680, 690 (specifically, The one or more incoupling optical elements are reflective polarizing optical elements). As shown, the incoupling optical elements 700, 710, 720 may also be disposed on the upper major surface of their respective waveguides 670, 680, 690 (or the top of the next lower waveguide). Well, in particular, those internally coupled optics are transmissive deflection optics. In some embodiments, the incoupling optical elements 700, 710, 720 may be located within the body of individual waveguides 670, 680, 690. In some embodiments, as discussed herein, the incoupling optical elements 700, 710, 720 selectively re-transmit one or more wavelengths of light while transmitting other wavelengths of light. It is directional and wavelength selective. Although illustrated on one side or at a corner of the individual waveguides 670, 680, 690, the internal coupling optics 700, 710, 720 may, in some embodiments, be the individual waveguides 670, 680, 690. It is to be understood that it may be arranged in other areas of.

  As shown, the incoupling optical elements 700, 710, 720 may be laterally offset from each other. In some embodiments, each incoupling optical element may be offset to receive light without its light passing through another incoupling optical element. For example, each incoupling optical element 700, 710, 720 may be configured to receive light from different image injection devices 360, 370, 380, 390, and 400, as shown in FIG. May be separated from other internal coupling optical elements 700, 710, 720 (eg, laterally spaced) so as to not substantially receive from other internal coupling optical elements 700, 710, 720. ..

  Each waveguide also includes an associated light dispersive element, eg, light dispersive element 730 is disposed on a major surface (eg, upper major surface) of waveguide 670 and light dispersive element 740 is a light guide element. The light dispersive element 750 is disposed on a major surface (eg, upper major surface) of the waveguide 680 and the light dispersive element 750 is disposed on a major surface (eg, upper major surface) of the waveguide 690. In some other embodiments, the light dispersive elements 730, 740, 750 may be disposed on the bottom major surface of the associated waveguide 670, 680, 690, respectively. In some other embodiments, the light dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of the associated waveguide 670, 680, 690, respectively, or the light The dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

  The waveguides 670, 680, 690 may be separated and separated by, for example, a gas, liquid, and / or solid layer of material. For example, as shown, layer 760a may separate waveguides 670 and 680 and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low index material (ie, a material having a lower index of refraction than the material forming the immediate vicinity of waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more, or 0.10 or less than the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b promote total internal reflection (TIR) of light through the waveguides 670, 680, 690 (eg, TIR between the top and bottom major surfaces of each waveguide). It may also function as a cladding layer. In some embodiments, layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include the nearest cladding layer.

  Preferably, the materials forming the waveguides 670, 680, 690 are similar or identical and the materials forming layers 760a, 760b are similar or identical for ease of manufacturing and other considerations. Is. In some embodiments, the material forming the waveguides 670, 680, 690 may differ between one or more waveguides, and / or the material forming layers 760a, 760b is still They may be different while retaining the various refractive index relationships mentioned above.

  With continued reference to FIG. 9A, rays 770, 780, 790 are incident on the set of waveguides 660. It should be appreciated that the light rays 770, 780, 790 may be launched into the waveguides 670, 680, 690 by one or more image launch devices 360, 370, 380, 390, 400 (FIG. 6). .

  In some embodiments, the light rays 770, 780, 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Each of the incoupling optical elements 700, 710, 720 deflects incident light such that the light propagates by TIR through a respective one of the waveguides 670, 680, 690. In some embodiments, each of the incoupling optical elements 700, 710, 720 transmits one or more particular wavelengths of light while transmitting another wavelength to the underlying waveguide and associated incoupling optical elements. Selectively deflect.

  For example, the incoupling optical element 700 may be configured to deflect a light ray 770 having a first wavelength or wavelength range while transmitting light rays 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. May be configured as. The transmitted light ray 780 impinges on and is deflected by the incoupling optical element 710, which is configured to deflect light of the second wavelength or range of wavelengths. Light ray 790 is deflected by an incoupling optical element 720 that is configured to selectively deflect light at a third wavelength or range of wavelengths.

  With continued reference to FIG. 9A, the deflected light rays 770, 780, 790 are deflected to propagate through the corresponding waveguides 670, 680, 690. That is, the incoupling optical element 700, 710, 720 of each waveguide deflects light into its corresponding waveguide 670, 680, 690 and incouples light into the corresponding waveguide. . Rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the individual waveguides 670, 680, 690 by TIR. The rays 770, 780, 790 propagate through the individual waveguides 670, 680, 690 by TIR until they strike the corresponding light dispersive elements 730, 740, 750 of the waveguide.

  9B, a perspective view of the multiple stacked waveguide embodiment of FIG. 9A is illustrated. As previously described, the incoupling rays 770, 780, 790 are respectively deflected by the incoupling optical elements 700, 710, 720 and then propagated by TIR in the waveguides 670, 680, 690, respectively. . Rays 770, 780, 790 then impinge on light dispersive elements 730, 740, 750, respectively. The light dispersive elements 730, 740, 750 deflect the light rays 770, 780, 790 to propagate toward the outcoupling optical elements 800, 810, 820, respectively.

  In some embodiments, the light dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPE deflects or disperses light into the outcoupling optical elements 800, 810, 820, and in some embodiments also as a beam of light of the present light as it propagates to the outcoupling optical elements. The spot size can be increased. In some embodiments, the light dispersive elements 730, 740, 750 may be omitted and the inner coupling optical elements 700, 710, 720 direct the light directly to the outer coupling optical elements 800, 810, 820. May be configured as. For example, referring to FIG. 9A, light dispersive elements 730, 740, 750 may be replaced with outcoupling optical elements 800, 810, 820, respectively. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EP) or exit pupil expanders (EPE) that direct light to the viewer's eye 210 (FIG. 7). It is understood that the OPE may be configured to increase the size of the eyebox in at least one axis and the EPE may increase the eyebox in an axis that intersects, eg, is orthogonal to, the axis of the OPE. I want to be done. For example, each OPE is configured to redirect a portion of the light that strikes the OPE into the EPE of the same waveguide, while allowing the rest of the light to continue to propagate through the waveguide. May be done. In response to the impact on the OPE, another portion of the remaining light is redirected back to the EPE and the remaining portion of that portion continues to propagate further down the waveguide or the like. Similarly, in response to the strike on the EPE, some of the impinging light is directed from the waveguide towards the user, and the rest of the light passes through the waveguide until it strikes the EP again. It continues to propagate, at which point another portion of the impinging light is directed from the waveguide, and so on. As a result, a single beam of internally coupled light is "replicated" each time a portion of that light is redirected by the OPE or EPE, thereby being cloned, as shown in FIG. Can form a beam field of light. In some embodiments, the OPE and / or EPE may be configured to modify the size of the beam of light.

  Thus, referring to FIGS. 9A and 9B, in some embodiments, the set of waveguides 660 includes, for each primary color, a waveguide 670, 680, 690 and an inner coupling optical element 700, 710, 720. It includes light dispersive elements (eg OPE) 730, 740, 750 and outcoupling optical elements (eg EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with a void / cladding layer between each one. The incoupling optics 700, 710, 720 redirect (or with different incoupling optics that receive different wavelengths of light) the incident light into its waveguide. The light then propagates into the individual waveguides 670, 680, 690 at an angle that will result in TIR. In the example shown, the light ray 770 (eg, blue light) is polarized by the first incoupling optical element 700 in the manner described above, and then continues to bounce through the waveguide to disperse the light dispersing element (eg, blue light). , OPE) 730, and then outcoupling optics (eg, EP) 800. Rays 780 and 790 (eg, green and red light, respectively) pass through waveguide 670, and ray 780 impinges on and is deflected by incoupling optical element 710. Ray 780 will then bounce through TIR through waveguide 680 to its light dispersive element (eg, OPE) 740 and then outcoupling optical element (eg, EP) 810. Finally, light ray 790 (eg, red light) passes through waveguide 690 and impinges on optical incoupling optical element 720 of waveguide 690. The light incoupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light dispersive element (eg, OPE) 750 and then to the outcoupling optical element (eg, EP) 820. Outcoupling optics 820 then finally outcouples light ray 790 to the viewer, who also receives the outcoupled light from other waveguides 670, 680.

  FIG. 9C illustrates a top-down plan view of the multiple stacked waveguide embodiment of FIGS. 9A and 9B. As shown, the waveguides 670, 680, 690 are vertically aligned with their associated light dispersive elements 730, 740, 750 and their associated outcoupling optical elements 800, 810, 820. May be done. However, as discussed herein, the incoupling optical elements 700, 710, 720 are not vertically aligned. Rather, the incoupling optical elements are preferably non-overlapping (eg, laterally spaced, as seen in the top and bottom views). As discussed further herein, the present non-overlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a 1: 1 basis, thereby allowing specific light sources to be specific. Allows to be uniquely coupled to the waveguide. In some embodiments, arrays that include non-overlapping, spatially separated, internally coupled optical elements can be referred to as a shift pupil system, where the internally coupled optical elements in these arrays correspond to sub-pupils. You can

  FIG. 9D illustrates an example of a wearable display system 60 in which the various waveguides and related systems disclosed herein can be integrated. In some embodiments, the display system 60 is the system 250 of FIG. 6, which schematically illustrates some portions of the system 60 in more detail. For example, the waveguide assembly 260 of FIG. 6 may be part of the display 70.

  With continued reference to FIG. 9D, display system 60 includes a display 70 and various mechanical and electronic modules and systems for supporting the functions of display 70. Display 70 may be coupled to frame 80, which is wearable by a display system user or viewer 90 and is configured to position display 70 in front of user 90's eye. The display 70 may be considered an eyepiece in some embodiments. In some embodiments, the speaker 100 is coupled to the frame 80 and is configured to be positioned adjacent to the ear canal of the user 90 (in some embodiments, another speaker not shown is also optional). Located adjacent to the user's other ear canal and may provide stereo / mouldable sound control). Display system 60 may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphone is configured to allow a user to provide input or commands to system 60 (eg, voice menu command selection, natural language question, etc.), and / or other Audio communication with a person (eg, another user of a similar display system) may be enabled. The microphone may also be configured as a peripheral sensor to collect audio data (eg, sound from the user and / or the environment). In some embodiments, the display system 60 is further configured to detect light, objects, stimuli, people, animals, places, or other aspects of the world around the user, one or more outward facing. May include an environment sensor 112 directed at. For example, the environmental sensor 112 may include one or more cameras, which are located outward facing to capture an image similar to, for example, at least a portion of the user's 90 normal field of view. Good. In some embodiments, the display system may also include a peripheral sensor 120a, which is separate from the frame 80 and on the body of the user 90 (eg, the head, torso, extremities, etc. of the user 90). May be attached to. Peripheral sensor 120a may be configured, in some embodiments, to obtain data characterizing a physiological condition of user 90. For example, the sensor 120a may be an electrode.

  With continued reference to FIG. 9D, the display 70 is operably coupled to the local data processing module 140 by a communication link 130, such as a wire or wireless connectivity, which is fixedly mounted to the frame 80 by a user. Fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 90 (eg, in a backpack configuration, in a belt-coupled configuration), etc. May be mounted in the above configuration. Similarly, the sensor 120a may be operably coupled to the local data processing module 140 by a communication link 120b, such as a wired conductor or wireless connectivity. The local processing and data module 140 may include a digital memory, such as a hardware processor and non-volatile memory (eg, flash memory or hard disk drive), both to assist in processing, caching, and storing data. May be used. Optionally, local processing and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may be a) sensors (image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein (eg, , Operably coupled to the frame 80, or otherwise attachable to the user 90) captured), and / or b) remote processing for passage to the display 70, possibly after processing or readout. It may include data obtained and / or processed using the module 150 and / or the remote data repository 160 (including data related to virtual content). The local processing and data module 140 is coupled such that these remote modules 150, 160 are operably coupled to each other and are available as resources to the local processing and data module 140, such as via a wired or wireless communication link, etc. Communication links 170, 180 may be operatively coupled to remote processing module 150 and remote data repository 160. In some embodiments, the local processing and data module 140 includes one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. May be included. In some other embodiments, one or more of these sensors may be attached to frame 80 or communicate with local processing and data module 140 via a wired or wireless communication path. It may be an independent structure.

With continued reference to FIG. 9D, in some embodiments the remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information, For example, it includes one or more central processing units (CPU), graphics processing units (GPU), dedicated processing hardware, and the like. In some embodiments, remote data repository 160 may comprise digital data storage facilities that may be available through the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, remote data repository 160 may include one or more remote servers that generate information, eg, augmented reality content, to local processing and data module 140 and / or remote processing module 150. Provide information to do so. In some embodiments, all data is stored and all calculations are done within the local processing and data modules, allowing full autonomous use from remote modules. At least part of a process (eg, generating image information, processing data) by an external system (eg, one or more processors, one or more computer systems), optionally including a CPU, GPU, etc. May be implemented to provide information to and receive information from modules 140, 150, 160, eg, via a wireless or wired connection.

I. Quality control based on depth information

  As described herein, a display system (eg, an augmented reality display system such as the display system 60 of FIG. 9D), according to various embodiments, can be used to monitor the user's eyes, eg, by monitoring the user's eyes. The dimensional fixation point may be determined. The fixation point is (1) x-axis (eg, lateral axis), (2) y-axis (eg, vertical axis), and (3) z-axis (eg, point depth, eg, from the user. It may indicate the location of the point in space along the depth. In some embodiments, the display system may utilize cameras, sensors, etc. to monitor the user's eyes (eg, the pupil of each eye, the cornea, etc.) and determine the line of sight of each eye. The line of sight of each eye can be generally understood to be a vector extending from the center of the retina of the eye through the lens of the eye. For example, the vector may generally extend from the center of the macula (eg, the fovea) through the lens of the eye. The display system may be configured to determine where the vector associated with the eye intersects, and this point of intersection may be understood to be the fixation point of the eye. In other words, the fixation point may be a place in the three-dimensional space where the user's eyes are converging. In some embodiments, the display system may filter slight movements of the user's eye, such as during fast movements (eg, saccades, microsaccades), such that the eye may occupy a location in three-dimensional space. The fixation point may be updated in response to determining the fixation. For example, the display system may be configured to ignore eye movements that are gazing at a point for less than a threshold duration.

  The resolution of the content presented by the display system, such as virtual objects or content, may be adjusted based on proximity to the fixation point, as discussed herein. It is to be appreciated that the display system may have access to, where information about the virtual object's location in three-dimensional space may be stored. The proximity of a given virtual object to the fixation point may be determined based on the known location of the virtual object. For example, the proximity between the virtual object and the fixation point is (1) the three-dimensional distance of the virtual object from the fixation point of the user, and (2) when the display frustum of the display system is divided into resolution adjustment zones. It may be determined by determining one or more of (3) the angular separation between the virtual object and the user's line of sight, relative to the resolution adjustment zone in which the virtual object is located. Virtual content closer to the fixation point may be presented with a higher resolution than content farther from the fixation point. In some embodiments, the resolution of the virtual content varies depending on the depth plane in which the virtual content is located with respect to the fixation point or the proximity of the depth plane in which the fixation point is arranged. In some embodiments, adjustments to resolution are made by a rendering engine, such as a rendering engine included in one or more graphics processing units, eg, one or more of modules 140, 150 (FIG. 9D). You may break.

  FIG. 10A illustrates an example of a top-bottom representation of a user viewing content (eg, content contained within display frustum 1004) presented by a display system (eg, display system 60, FIG. 9D). The representation includes the eyes 210, 220 of the user and the determination of the fixation point 1006 of the eyes 210, 220. As shown, the line of sight of each eye is represented as a vector (eg, vector 1003A, 1003B), and the display system may, for example, determine where those vectors converge in front of the eyes 210, 22. , Fixation point 1006 is detected. In the illustrated example, the fixation point 1006 corresponds to the location of the first virtual object 1008A presented by the display system. Examples of systems and methods for eye tracking are described in US patent application Ser. No. 14 / 690,401, filed Apr. 18, 2015 (incorporated by reference for all purposes) and the accompanying Can be found in the Appendix. For example, eye tracking systems and methods are described at least in Figures 25-27 of the Appendix and, at least in part, as described herein, for eye tracking and / or determining fixation points. Can be used to:

  With continued reference to FIG. 10A, the second virtual object 1008B is also presented by the display system in the display frustum 1004. A view of these virtual objects 1008A, 1008B as seen by a viewer is shown in rendered frame 1010. The rendered frame 1010 may include a first virtual object 1008A rendered at a first resolution, while a second virtual object 1008B located away from the fixation point 1006 has a second lower resolution. Rendered with. Specifically, it may be determined that the second virtual object 1008B is at a deeper depth than the first virtual object 1008A and is positioned toward the side surface thereof. For example, the display system may determine the depth of the second virtual object 1008B, as discussed herein, or, optionally, the content provider associated with the virtual content determines that the display system is virtual. It may indicate the depth of the virtual object that may be used to render the object. Thus, the fixation point 1006 describes the three-dimensional location in space that the user is looking at, as described above, and the second virtual object 1008B is located at a depth further from the user and It can be determined that it is displaced laterally from 1006.

  Without being limited by theory, with the user's eyes 210, 220 looking at the first virtual object 1008A, an image of the first virtual object 1008A may enter the fovea of the user, while The image of the second virtual object 1008B is considered not to enter the fovea. As a result, the second virtual object 1008B has reduced resolution without a significant impact on the perceived image quality of the display system due to the lower sensitivity of the human visual system to the second virtual object 1008B. May be done. In addition, the lower resolution advantageously reduces the computational load required to provide the image. As discussed herein, the second virtual object 1008B is rendered, the resolution may be based on proximity to the fixation point 1006, and a reduction in resolution (eg, of the first virtual object 1008A). (With respect to resolution) may increase with decreasing proximity (or increasing distance) between fixation point 1006 and virtual object 1008A. In some embodiments, the rate of decrease in resolution may be based on the rate of decrease in the density of cones in the human eye or the loss of visual acuity with distance from the fovea.

  It should be appreciated that the resolution of the various virtual objects presented by the display system may change dynamically as the fixation point changes location. For example, FIG. 10B illustrates another example of a top-bottom representation of a user viewing content presented by a display system. As illustrated in FIG. 10B, the user is now focused on the second virtual object 1008B, as compared to FIG. 10A where the user was focused on the first virtual object 1008A. By monitoring the user's gaze 1003A, 1003B, the display system determines that the eyes 210, 220 are congested on the second virtual object 1008B and sets the location as a new fixation point 1006.

  In response to detecting this change in the location of fixation point 1006, the display system now renders the second virtual object 1008B a higher resolution than the first virtual object 1008A, as shown in rendered frame 1010. Render with. Preferably, the display system monitors the user's gaze 1003A, 1003B at a sufficiently high frequency so that the transitions in the resolution of the first virtual object 1008A and the second virtual object 1008B are substantially imperceptible to the user. In addition, the resolution of the virtual object is changed sufficiently quickly.

  FIG. 10C illustrates another example of a top-bottom representation of a user viewing content via a display system (eg, display system 60, FIG. 9D). In the illustrative example, the user's field of view 1004 is illustrated with fixation point 1006. Three virtual objects are shown, with the first virtual object 1012A being closer to the fixation point 1006 than the second virtual object 1012B or the third virtual object 1012C. Similarly, the second virtual object 1012B is shown closer to the fixation point 1006 than the third virtual object 1012C. Thus, when virtual objects 1012A-1012C are presented to the user, the display system is provided with a step of rendering the first virtual object 1012A with a greater resource allocation than the second virtual object 1012B (eg, object 1012A). , But the second virtual object 1012B receives a larger resource allocation than the third virtual object 1012C. The third virtual object 1012C may optionally be outside the field of view 1004 and therefore not rendered at all.

  The resolution adjustment zone is illustrated in the example of FIG. 10C, where the zone is elliptical (eg, circular) described along the depth and lateral axes. As shown, the fixation point 1006 is inside the central zone 1014A and the first virtual object 1012A extends between the zones 1014B, 1014C and within the foveal cone 1004a of the user. The first virtual object 1012A may thus be presented to the user at the resolution associated with the zone 1014B or 1014C, or optionally, a portion of the object 1012A within the zone 1014B may be presented according to the resolution of the zone 1014B. The rest of the zone 1014C may be presented according to the resolution of the zone 1014C. For example, in some embodiments where the zone is assigned a reduced resolution from the maximum (eg, highest) resolution, the first virtual object 1012A may be presented at the assigned resolution. Optionally, the first virtual object 1012A is programmed to display at a resolution (eg, the display system is at the highest resolution associated with any zone across which the first virtual object 1012A extends). May be present) or a measure of the central tendency of resolution (eg, the measure may be weighted according to the extent to which object 1012A is located within zones 1014B, 1014C). With continued reference to FIG. 10C, it should be appreciated that the resolution adjustment zones at different distances from the fixation point 1006 may have different shapes. For example, zone 1014C may have a different shape than zones 1014A-1014C and conform to the contour of field of view 1004. In some other embodiments, one or more of zones 1014A-1014C may have a different shape than one or more of zones 1014A-1014C.

  FIG. 10D is a block diagram of an exemplary display system. The exemplary display system (eg, display system 60, FIG. 9D) may be an augmented reality display system and / or a mixed reality display system, which, as described herein, has a user's perspective. Accordingly, the usage of rendering hardware resources can be adjusted. For example, as described above with respect to FIG. 10C, rendering hardware resources 1021 can be adjusted according to the user's fixation point. A resource arbiter 1020 may be implemented to coordinate the use of such resources 1021; for example, the arbiter 1020 may direct the resource 1021 to a particular application process 1022 associated with presenting a virtual object to a user. Can be distributed. Resource arbiter 1020 and / or rendering hardware resources 1021 are optionally included within local processing and data module 140 of display system 60 (eg, as shown in FIG. 9D) and / or remote processing module 150. Good. For example, rendering hardware resource 1021 may comprise a graphics processing unit (GPU), which may be included in module 140 and / or module 150, as described above with respect to FIG. 9D.

  As an example of adjusting resource 1021, with respect to FIG. 10C, a first virtual object 1012A associated with a first application process has more resources 1021 than a second virtual object 1012B associated with a second application process. Can be apportioned. The virtual objects associated with the application process 1022 are rendered based on the allocated resources 1021 and included in the frame buffer 1024 that will be composited (eg, by the compositor 1026) into the final frame buffer 1028. be able to. The final frame buffer 1028 can then be presented by the display hardware 1030, eg, the display 70 illustrated in FIG. 9D, and the rendered virtual object is resolution adjusted.

  As disclosed herein, the resolution of the virtual object may be determined based on the proximity of the virtual object and the fixation point. In some embodiments, the resolution may be modified as a function of the distance between the virtual object and the fixation point. In some embodiments, the modification may occur in discrete steps. That is, similar modifications may be applied to all virtual objects located within a particular volume or zone. FIG. 11A1 illustrates an example of a top-down representation of adjustments at resolutions within different resolution adjustment zones based on 3D fixation tracking. The display system may divide the display frustum into a plurality of volume or resolution adjustment zones and modify the resolution in discrete steps corresponding to these zones. Thus, in some embodiments, in order to determine the adjustment in the resolution of the virtual content, the display system provides information describing the volume of space (hereinafter referred to as the resolution adjustment zone) and each space of the resolution adjustment. Allocation to volume may be used. As shown, the field of view provided by the display system (eg, the display frustum of the display) is separated into a plurality of different zones, each of which spans a range of depth from the user (eg, depth range 1102A-1102E). Include. In some embodiments, each depth range 1102A-1102E has a single associated depth plane that can be presented by the display system. With continued reference to FIG. 11A1, five zones are contiguous along the lateral direction, encompassing each identified depth range from the user. In the exemplary top and bottom views shown, the field of view is divided into a grid 1100 of 25 zones. Each zone represents a volume of real world space in which virtual content may be installed for a user.

  The zones may also extend in the present vertical direction (eg, along the y-axis, not shown), so that the illustrated grid 1100 may be understood to represent one cross-section along the vertical direction. Please understand that. In some embodiments, multiple zones are also provided in the vertical direction. For example, there may be 5 vertical zones per depth range, or a total of 125 resolution adjustment zones. An example of such a zone extending in three dimensions is illustrated in FIG. 11B and described below.

  With continued reference to FIG. 11A1, the user's eyes 210, 220 gaze at a particular fixation point 1006 within the grid 1100. The display system may determine the location of fixation point 1006 and the zone in which fixation point 1006 is located. The display system may adjust the resolution of the content based on the proximity of the virtual content and the fixation point 1006, which may include determining the proximity of the zone in which the virtual content and the fixation point 1006 are located. Good. As an example, for content contained within the zone in which fixation point 1006 is located, the resolution may be set to a particular polygon count, in the example 10,000 polygons. Based on the distance from the fixation point 1006, the content contained in the remaining zones may be adjusted accordingly. For example, content contained within a zone adjacent to the zone containing fixation point 1006 may be rendered at a lower resolution (eg, 1,000 polygons). The example of FIG. 11A1 illustrates, as an example, the step of adjusting the polygon count, as described herein, although the step of adjusting the resolution may be other than the resolution of the presented content. The step of making a correction may be included. For example, adjustments in resolution include adjusting polygon counts, adjusting primitives used to create virtual objects (eg, adjusting the shape of primitives, eg, changing a primitive from a triangular mesh to a quadrilateral mesh). Etc.), adjusting the actions performed on the virtual object (eg, shader actions), adjusting texture information, adjusting color resolution, or depth, the number of rendering cycles or frame rate. And one or more of adjusting quality at one or more points in a graphics pipeline of a graphics processing unit (GPU).

  In addition, the example of FIG. 11A1 provides a specific example of the difference in polygon counts within different resolution adjustment zones, but other absolute numbers of polygons and other resolutions with distance from fixation point 1006. The rate of change is also considered. For example, the resolution drop from the fixation point 1006 may be based on a descent rate that is symmetrical about the depth and lateral distance from the fixation point 1006, although other descent relationships may also be utilized. For example, the lateral distance from the fixation point 1006 may be associated with a greater drop in resolution with respect to the depth distance from the fixation point 1006. Further, the size of each zone contained within the grid (eg, the size of the volume of space in the zone) may optionally be different (eg, the zones may vary radially from the foveal axis). .. In some embodiments, the descent may continue from fixation point 1006 such that a resolution relationship with a discrete zone having an assigned resolution or a zone containing fixation point 1006 is not utilized. For example, the descent from the fixation point 1006 to a particular zone 1108 (eg, the zone in which the content is rendered at a resolution of 100 polygons) is continuous from the fixation point 1006 to the edge of the grid (eg, the edge of the specific zone 1108). It may be modified to be a descent. It should be appreciated that each of the above considerations also applies to the vertically extending zones.

  In some embodiments, the number and size of zones included in the grid may be based on the confidence level associated with the determination of the user's fixation point 1006. For example, the confidence level may be based on the amount of time the user's eye was gazing at the fixation point 1006, with a shorter amount of time associated with a lower confidence level. For example, the display system may monitor the user's eyes at a particular sampling rate (eg, 30 Hz, 60 Hz, 120 Hz, 1 kHz), and the continuous samples indicate that the user generally maintains a fixation point 1006. , The reliability at the fixation point 1006 can be increased. Optionally, a particular threshold of fixation may be utilized, for example, fixation at the same or similar fixation point over a specific duration (e.g., 100-300 ms) may be considered reliable. While less than a certain duration may be associated, less confidence may be associated. Similarly, intraocular variations such as mydriasis, which can affect the user's fixation point determination, can reduce the reliability of the display system. It should be appreciated that the display system may use sensors such as a camera imaging device (eg, camera assembly 630, FIG. 6) to monitor the eye. Optionally, the display system may utilize a combination of sensors to determine the eye gaze of the user (eg, an infrared sensor utilized to detect infrared reflections from the eye and identify the pupil, Different eye gaze determination processes may be utilized, such as a visible light imaging device utilized to detect the iris of the eye). The display system may increase the confidence level when the eye gaze determination processes are in agreement and may reduce the confidence level when they are inconsistent. Similarly, for display systems that perform a gaze determination process for only one eye, each gaze determination process may be associated with a particular confidence level (eg, one decision process is considered more accurate than the other). Obtained), the size of the resolution adjustment zone may be selected based, at least in part, on the process being implemented.

  In some embodiments, the display system may increase or decrease the number of zones with each update of the fixation point 1006. For example, as the confidence associated with fixation point 1006 increases, more zones may be utilized, and as the confidence decreases, fewer zones may be utilized. FIG. 11A2 illustrates an example of a top and bottom representation of the resolution adjustment zone at different times as the size and number of zones change. At time t = 1, the user's field of view may be divided into an initial set of zones, as seen in the top and bottom figures. At time t = 2, the confidence at the location of fixation point 1006 increases and the display system may also decrease the size of the zone occupied by fixation point 1006 and rendered at high resolution. Optionally, as shown, the size of other zones may also be reduced. At time t = 3, the confidence at the location of fixation point 1006 decreases and the display system may also increase the size of the zone occupied by fixation point 1006 and rendered at high resolution. Optionally, the size of other zones may also be increased, as shown. It should be appreciated that multiple zones may also extend in the y-axis and similar increases or decreases in zone size and number may also be established on that axis. For example, the size of a zone extending vertically on the y-axis may decrease with increasing confidence, while the size may increase with decreasing confidence. Optionally, the display system may determine the confidence level of fixation point 1006 for each frame presented to the user by the display system, where t = 1, t = 2, and t = 3 represent different frames. May be. Assigning more zones may require an increase in calculated power (eg, a display system may need to adjust the resolution of more content, identify zones that contain content, etc.) , The display system may balance the increase in required calculated power provided by the increase in the number of zones with the savings in calculated power provided by the potential decrease in content resolution.

  Referring again to FIG. 11A1, the grid can change dynamically in the sense that the fixation point 1006 can be set to be located at the center of the grid (eg, the center of gravity). Therefore, the display system may avoid the edge case, where the fixation point 1006 is determined to be located on the apex of the grid. For example, as the user's eyes rotate and then gaze at different three-dimensional locations in space, the grid may also move with the user's line of sight.

  11B-11E illustrate examples of various resolution adjustment zone configurations. Additional shapes and configurations of resolution adjustment zones, not shown, may be utilized and the embodiments should not be considered to be comprehensive. In addition, in some figures, the user's eyes 210, 220 may be shown separated from the various resolution adjustment zones for ease of illustration and clarity. With respect to all these figures, it should be appreciated that the eyes 210, 220 may be located at or within the boundaries of the zones (see, eg, FIG. 11A1).

  FIG. 11B illustrates an example of a three-dimensional representation of a portion of the resolution adjustment zone of FIG. 11A1. 11A1 may be understood to illustrate a cross-sectional view taken along the plane 11A1-11A1 of the three-dimensional representation of FIG. 11B, and FIG. 11B illustrates one of the resolution adjustment zones of FIG. Please note that some of the are omitted. With continued reference to FIG. 11A1, the field of view provided by the display system is separated into 27 zones. That is, the field of view is divided into three depth ranges 1102B-1102D, each depth range including a 3 × 3 grid of zones extending laterally and vertically in the depth range.

  The determined fixation point 1006 is illustrated as being in a zone located in the center of the field of view. Virtual objects located in a zone outside the zone containing the fixation point 1006 may have reduced resolution according to distance from the fixation point 1006 zone, as discussed herein. Since the zones extend laterally and vertically, the reduction in resolution is at distances on the lateral, vertical, and depth axes (x, y, and z-axis, respectively) from the fixation point's resolution adjustment zone. Can occur based on. For example, in some embodiments, virtual objects located in zone 1108 can have reduced resolution according to lateral distance, as shown in FIG. 11A1 (eg, zone 1108 includes fixation point 1006). Zone and the same vertical portion of the user's field of view, and may be on the same depth plane).

  Similar to the above, and similarly to the zones described below in FIGS. 11C-11E, the user's fixation point can optionally be maintained to be located at the center (eg, center of gravity) of the zone, Alternatively, the zones can be fixed with respect to the user's field of view and the user's fixation point can be located within any of the zones.

  FIG. 11C illustrates another example of a configuration for the resolution adjustment zone. In the example, the fields of view provided by the display system are each illustrated as being separated into elliptical zones that encompass a particular three-dimensional volume of space. Similar to FIG. 11A1, each zone (eg, zones 1112A-112D) extends along the lateral and depth dimensions. In some embodiments, each zone also extends to encompass at least a portion of the user's vertical field of view. The fixation point 1006 is illustrated as being at the center of the zone (eg, within zone 1112A). Virtual objects located within a zone outside zone 1112A may have reduced resolution according to distance from zone 1112A, for example, according to the techniques described herein. For example, each zone outside zone 1112A can be assigned a particular resolution, or the rate of descent can be utilized to determine the reduction in resolution. Zone 1112D is illustrated as the zone furthest from zone 1110A, and the reduction in resolution may be greatest within zone 1112D.

  FIG. 11D illustrates an example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C, and FIG. 11C shows a cross-sectional view taken along the plane 11C-11C. In this example, the fields of view provided by the display system are each illustrated as being separated into ellipsoidal zones that encompass a three-dimensional volume of space. The user's fixation point 1006 is illustrated in the center of gravity of the user's field of view and is located within zone 1112A. Optionally, FIG. 11D may represent each ellipse of FIG. 11C transformed into an ellipsoid. In some embodiments, the size of zone 1112A of FIG. 11C along the depth and lateral directions can define the size of the major axis of zone 1112A of FIG. 11D along the X and Z axes. The various zones may form concentric spheres or ellipsoids.

  11E illustrates another example of a three-dimensional representation of the resolution adjustment zone of FIG. 11C, and FIG. 11C illustrates a cross-sectional view taken along the plane 11C-11C. The field of view provided by the display system is illustrated as being separated into stacked levels of similar concentric zones. For example, FIG. 11E can represent how the ellipse of FIG. 11C is extended along a vertical direction to create a cylinder. The cylinders may then be vertically separated such that each cylinder covers a portion of the user's vertical field of view. Therefore, FIG. 11E illustrates nine zones of cylindrical shape. Each zone additionally excludes any interior zone (eg, ellipsoid 1112B will exclude the volume of space encompassed by ellipsoid 1112A, will include the volume of space). In an example, fixation point 1006 is illustrated as being within central zone 1110A, and virtual objects located outside central zone 1110A may have reduced resolution in accordance with the techniques described herein. it can.

  FIG. 12A illustrates a flowchart of an exemplary process 1200 for adjusting the resolution of content according to proximity with a 3D fixation point. For convenience, the process 1200 may be a display system (eg, wearable display system 60, which may include processing hardware and software, and optionally sends information to one or more computers or other external processing systems. May be provided, for example, processing may be offloaded to an external system and information may be received from the external system).

  At block 1202, the display system determines the user's 3D fixation point. As explained above, the display system may include sensors to monitor information associated with the user's eyes (eg, eye orientation). The non-exhaustive list of sensors includes infrared sensors, ultraviolet sensors, visible wavelength light sensors. The sensor may optionally output infrared, ultraviolet, and / or visible light on the user's eye to determine the reflection of the output light from the user's eye. As an example, infrared light may be output by an infrared light emitter and an infrared light sensor. It should be appreciated that the sensor, which may include a light emitter, may correspond to the imaging device 630 of FIG.

  The display system utilizes sensors to determine the line of sight associated with each eye (eg, a vector extending from the user's eye such as extending from the fovea through the lens of the eye) and the line of sight of each eye. You may. For example, the display system may output infrared light onto the user's eye and the reflection from the eye (eg, corneal reflection) may be monitored. The vector between the eye's pupil center (eg, the display system may determine the center of gravity of the pupil, eg, through infrared imaging) and the reflection from the eye is used to determine the eye's line of sight. You may. The line of sight intersection may be determined and assigned as a three-dimensional fixation point. The fixation point may thus indicate where the content will be rendered at full or full resolution. For example, based on the determined line of sight, the display system may triangulate a three-dimensional location in the space in which the user is gazing. Optionally, the display system may utilize orientation information associated with the display system (eg, information describing the orientation of the display system in three-dimensional space) when determining the fixation point.

  At block 1204, the display system obtains location information associated with the content that is being or will be presented to the user by the display system. Prior to rendering the content for presentation to the user (eg, via the output of the waveguide, as described above), the display system determines the content to be presented to the user. You may acquire the location information linked | related with. For example, as described above, virtual content may be presented to a user such that the content appears to be located in the real world (eg, the content may be located at different depths within the user's field of view). May be). It should be appreciated that the display system may include, or have access to, a three-dimensional map of the present environment, which may signal the location of any virtual content within the environment. Referring to this map, the display system accesses information defining the three-dimensional location of the virtual content within the user's field of view (eg, within the display frustum, as illustrated in FIGS. 10A-10B), You may provide it.

  At block 1206, the display system adjusts the resolution of the virtual content that will be displayed to the user. The display system adjusts the resolution of the content based on its proximity to the 3D fixation point. A rendering engine, such as a rendering engine implemented by a processing device (eg, a central processing unit, a graphics processing unit), that renders content for presentation to a user, allocates the resources invested in rendering the content. May be adjusted (eg, the rendering engine may adjust the resolution of the content).

  The display system may determine a distance in a three-dimensional space between the content to be presented to the user and the user's fixation point, and based on the determined distance, reduce the resolution of the content. Good. The reduction may be determined according to a continuous function that correlates the resolution of the content with the rate of descent, for example, the display system may obtain the resolution and render the content based on the continuous function. Optionally, the display system may determine the distance from the center of gravity of the content to the fixation point and may render the content at a certain resolution based on the distance. Optionally, the display system may render portions of the same content at different resolutions according to the distances of the various portions to the fixation point (eg, the display system may separate the content into portions, closer to each other). The farther part may be rendered with reduced resolution compared to the part).

  In some embodiments, the display system may access information that may be used to separate a user's field of view (eg, corresponding to a display frustum) into zones, each zone containing content. Represents the volume of space obtained. The information accessed, eg, the grid illustrated in FIG. 11A1, may indicate a particular resolution to utilize when rendering the content that will be contained within each zone. Set at the center of the grid. In addition, the grid may show the rate of descent in resolution for use when rendering the content. For content contained in multiple zones (eg, content located in the three-dimensional space required by the two zones), the display system optionally adjusts the resolution of the content to accommodate a single zone. Or optionally, the resolution of the content may be adjusted according to the corresponding zone in which the portion is located.

  When setting the content resolution, the display system renders the content located at the fixation point (eg, in the same zone as the fixation point) at full or full resolution. The maximum resolution ensures that the content is presented to the user at a refresh rate above a threshold (eg, 60 Hz, 120 Hz), and optionally, the content exceeds the congestion / divergence motion rate (eg, above 60 ms). The display system hardware and / or software is capable of rendering while ensuring that it is updated at a faster rate than the accommodation time (eg, 20 ms to 100 ms) and reduces the perceptibility of changes in resolution. There may be a maximum value. The display system may dynamically modify the maximum resolution based on the display system's available resources, for example, prior to the display system rendering each frame. For example, as more content is to be presented to the user, the maximum resolution of the content is reduced and the display system desires frames of rendered content to reduce the perceptibility of changes in resolution. It may be ensured that the rate of presentation may be exceeded. The display system may optionally monitor the frames per second at which the content is being presented and adjust the maximum resolution and / or the rate of resolution drop based on the distance from the fixation point to be presented. It may be ensured that the frames per second do not fall below the threshold rate. As an example, the display system may render the content, such as the first virtual object located in the fixation zone, at full resolution. Instead of reducing the maximum resolution of the first virtual object and ensuring that frames per second remain above a certain threshold, the display system dynamically scales the rate of resolution decrease based on distance. May be increased. In this way, the display system may adjust the resolution assigned to each zone outside the fixation zone. Optionally, the display system may set a minimum resolution that may be used within each zone outside the fixation zone and may adjust the maximum resolution if the minimum resolution would be exceeded (eg, , The display system may reduce the maximum resolution if it needs to reduce the resolution of the content below a minimum value and maintain a threshold rate). Similarly, the display system may reduce the maximum resolution without reducing the resolution of content in zones outside the fixation zone. Optionally, the user of the display system may indicate whether content located near the fixation point prefers to be favored over other content.

  In some embodiments, the display system optionally utilizes the angular proximity of the content and the user's line of sight to adjust the resolution of the content, as described in more detail below with respect to FIGS. 13-14. Good. For example, if the particular content is outside the zone in which the fixation point is located, but within the threshold proximity of the user's line of sight, such that the particular content would enter the fovea of the user's eye, The display system may cause certain content to be rendered at a higher resolution (eg, maximum resolution or greater than that shown in the grid illustrated in FIG. 11A1). Optionally, the display system may reduce the resolution of the particular content and apply a blurring process (eg, Gaussian blur) to the particular content. In this way, the particular content may be rendered blurry while being rendered at a lower resolution, indicating that the particular content is further from the user than the fixation point, for example. In addition, blurring may reduce the perceptual power of lower resolution (eg, blurring may reduce the perceptual power of increased pixel size due to lower resolution).

  Exemplary operations associated with presenting virtual content are illustrated in FIGS. 12B-12C (eg, a rendering pipeline). In the example of FIG. 12B, a three-dimensional scene is presented to the user without any adjustment to resolution made as described herein. In FIG. 12C, adjustments to resolution are performed according to fixation point information, as described herein. For example, one or more of the following adjustments: reducing vertex motion complexity, reducing planar fill level of detail, reducing geometry generation, pixel motion of multiple pixels. Steps such as reducing complexity / aggregation can be implemented. The adjustment may advantageously be performed at different steps in the pipeline for presenting virtual content, as shown, and may depend on the particular software and / or software utilized to present the virtual content. It can be optimized according to the hardware. It should be appreciated that the fidelity zone described in Figure 12C is a resolution adjustment zone.

  Referring again to FIG. 12A, the display system presents the adjusted content to the user at block 1208. As explained above, the display system adjusts the resolution of the content based on the proximity to the 3D fixation point. The display system then presents the rendered content to the user at the associated location. In some embodiments, the display system may perform process 1200 for each frame of content to be rendered, or adjust the resolution of the content as the user adjusts its fixation point. Good.

  As mentioned above, in some embodiments virtual objects may be presented at different depths while still in line-of-sight of the user. FIG. 13 illustrates an example of a representation of a user viewing a plurality of virtual objects that are aligned with the user's line of sight. The exemplary representation includes the line of sight 1003A, 1003B of the user's eye 210, 220 along with the user's field of view (eg, display frustum 1004 of the display system), which is fixed at the fixation point on the first virtual object 1008A. To be seen.

  As shown, the second virtual object 1008B may be such that the second virtual object 1008B will enter the fovea of the user (eg, enter at least one fovea of any of the eyes). It is within the angular proximity of the user's line of sight (eg, one or both of the line-of-sight vectors 1003A, 1003B). For example, in response to rendering frame 1110, second virtual object 1008B is located behind (eg, deeper perceived depth from) first virtual object 1008A. It should be understood that the fovea is the part of the retina that has the highest visual acuity. Since the second virtual object 1008B will enter the fovea of the user, the resolution of the second virtual object 1008B is reduced (eg, at least as described above with respect to FIG. 11A1). ), The user may perceive a reduction in resolution. To avoid a perceptible reduction in resolution, the display system causes (1) the second virtual object 1008B to render at the same resolution as the first virtual object 1008A or within the threshold resolution of the first virtual object 1008A. , And / or (2) render the second virtual object 1008B with reduced resolution (eg, as shown in FIG. 11A1) and apply blur to the second virtual object prior to presentation to the user. You may let me. Without being limited by theory, blur may mask the reduction in resolution while providing depth cues.

  FIG. 14 is a flow chart of an example of a process 1400 for adjusting virtual content based on an angular distance from a user's line of sight. For convenience, the process 1400 may include a display system (eg, a wearable display system 60, which may include processing hardware and software, and, optionally, information that is external to one or more computers or other processing units. May be provided to an external system and information may be received from the external system). In the exemplary process 1400, the display system is a variable focus display system in which each frame is presented on the same depth plane, optionally convolving all content to be presented into a single frame buffer. That is, the variable focus display system presents virtual content on one depth plane at a time.

  The display system determines the user's 3D fixation point (block 1402) and obtains location information associated with the content to be presented (block 1404). Blocks 1402 and 1404 may correspond to blocks 1202 and 1204 of FIG. 12A, respectively. As described above with reference to FIG. 12A, the display system monitors the user's eye movement (eg, eye orientation) and determines the user's fixation point. The display system may obtain location information (eg, in the next frame) of the content to be presented, and then adjust the resolution of the content.

  With continued reference to FIG. 14, the display system will have reduced resolution and determines content that is located within a threshold angular distance from the user's line of sight (block 1406). The display system will have reduced resolution due to the proximity of the content from the fixation point (eg, the content is located deeper than the fixation point) but will enter the fovea of the user (eg, the user). Content that would be within a threshold angle from the user's line of sight). Since the content will enter the fovea of the user, the user may be able to perceive a reduction in resolution, such as with the three-dimensional fixation-point foveated rendering described herein. It should be appreciated that the content block 1406 may include implementing the blocks illustrated in FIG. 12C, particularly the blocks identified in section “GPU”.

  As a result, at block 1408, the display system may optionally cause the determined content to be rendered at a higher resolution. The display system is reduced to be at full resolution (eg, the same resolution as the fixation point or content located in the same zone or volume of space as the fixation point, or otherwise assigned to the content). The resolution of the determined content may be adjusted to exceed the resolution (eg, as described in block 1406).

  At block 1410, the display system may optionally reduce the resolution of the content and may blur the content prior to presentation to the user. As explained above, variable focus display systems may utilize a single display buffer to present content to the user. Since the variable focus display system presents all content in the same depth plane, the variable focus display system may utilize the same display buffer to output the content from, for example, a rendering engine.

  Optionally, the display system may utilize an initial depth buffer, where each depth buffer is assigned one or more depth planes, and may combine the initial depth buffers to obtain the display buffer. Referring to the illustration of FIG. 13, the first depth buffer may include a first virtual object 1306, while the second depth buffer may include a second virtual object 1308. The display system may then apply the blurring process to the second depth buffer or specific content contained within the second depth buffer (eg, the display system may apply the blurring process to the second virtual content 1308). Apply, but need not apply to other content that is on the same depth plane but at a greater angular distance from the user's line of sight). After performing the blurring process, the display system combines the first depth buffer and the second depth buffer (eg, the display system adds occlusion and is invisible, eg, due to occlusion by the first virtual object 1306). Part of the second virtual object 1308 may be removed), and the display buffer may be obtained.

  An exemplary blurring process may include the display system performing a convolution of the kernel associated with the blurring (eg, a Gaussian kernel, a circular kernel such as to reproduce a destructive effect, a box blur, etc.) on the content. In this way, the reduction in resolution may be masked, while the processing savings resulting from reducing resolution may be maintained. Optionally, the intensity (eg, the degree to which the content is blurred) associated with the blurring process is also based on the difference in depth between the user's fixation point and the content and / or the angular proximity of the content to the user's line of sight. Good. For example, the degree of blurring may increase as the degree of proximity to the user's line of sight increases.

  In some embodiments, the display system may utilize the features of blocks 1408 or 1410 according to the display system hardware and / or software. For example, certain hardware (eg, a graphics processing unit) may be able to perform the blurring process in hardware without reaching a threshold on the performance of the hardware. With this particular hardware, the display system may be configured to reduce the resolution of the content and then blur the content. However, other hardware may be slow to perform the blurring process and rendering the content at a higher resolution may result in higher performance. With respect to this other hardware, the display system may be configured to render the content at a higher resolution. Furthermore, the decision whether to render the content at higher resolution or lower resolution with blurring may depend on the type of content to be displayed. For example, the display system may be configured to render text at a higher resolution while rendering shapes at a lower resolution and blur.

With continued reference to FIG. 14, at block 1412, the display system presents the content to the user. The display system may present the adjusted content to the user from the same display buffer, for example, as described above.

II. Resolution adjustment based on ambient lighting level

  In addition to, or as an alternative to, a reduction in resolution along the z-axis, various other schemes for presenting virtual content with a reduction in resolution may be implemented in some embodiments. .. Advantageously, some aspects of the virtual content may be presented with a relatively high resolution and some other aspects may be presented with a relatively low resolution, as described herein. Often, this may reduce the calculation and energy resource usage by the display system, while preferably having little impact on the perceived image quality of the virtual content.

  Referring now to FIG. 15, an example of a representation of the retina of the user's eye is illustrated. The view shown shows a retina 1500 as seen when viewed head-up along the visual axis of the retina. Retina 1500 includes fovea 1510 surrounded by peripheral area 1530. Within fovea 1510 is fovea 1520, which intersects the visual axis.

  It is to be understood that the retina contains two types of photoreceptors, rods and cones. In addition, the distribution of these photoreceptors across the retina fluctuates, providing different rod and cone densities across the retina.

  Referring now to FIG. 16, an example of the retina 1500 resolution and rod and cone densities across it of FIG. 15 is schematically illustrated. The x-axis shows the eccentricity with respect to the point where the visual axis intersects the retina. The right direction on the page is the nose direction, and the left direction on the page is the temple direction. As shown, the resolution of the human eye roughly correlates with the density of photoreceptors (rods and cones) within the retina. As a result, in some embodiments, the reduction or taper in the resolution (eg, spatial resolution) of the virtual content on the x and y-axes (eg, on a given depth plane) results in cone density, rod density. , Or may substantially follow the reduction of rod and cone density clusters across the retina. For example, the tendency for resolution reduction with distance from a fixation point across the user's field of view is to show photoreceptor density (eg, cone density, rod density, or a collection of rod and cone densities) over a corresponding portion of the retina. ) Change within ± 50%, ± 30%, ± 20%, or ± 10%. In some embodiments, the reduction in resolution with distance from the fixation point is gradual and substantially follows the density change. In some other embodiments, the reduction in resolution may occur in steps (eg, one step, two steps, etc.). For example, there may be two steps: the highest resolution region of the field of view that is correlated with the foveola, the medium resolution region that is correlated with the fovea, and the lower resolution region that is correlated with the peripheral area.

  With continued reference to FIG. 16, it should be understood that different photoreceptors have different levels of activity under different light conditions, eg, different ambient lighting levels. As a result, the reduction in resolution that follows the density of photoreceptors may not be noticeable to the user at some lighting levels, but may be noticeable at other lighting levels. Considered as sex. As a result, in some embodiments, the reduction in resolution of virtual content along the x, y, or z-axis may be set with reference to external light conditions.

For example, the visual behavior of the eye can be divided into three modes based on light conditions. The three modes are photopic, mesopic, and scotopic. Photopic typically bright conditions, including, for example, about 10~10 ⁸ cd / ^{m 2,} occurs at about 3 cd / ^{m 2} or more ambient light or illumination level. In photopic the cones are predominantly active. In scotopic vision, the rods are predominantly active. In mesopic vision, both rods and cones can be active. As used herein, ambient light conditions or lighting levels refer to the amount of light that the user's eye and its retina are exposed to.

Mesopic vision typically occurs under lower light conditions, eg, illumination levels of about 10 ⁻³ to 10 ^0.5 cd / m ² . Both cones and rods are active at at least some illumination levels within mesopic vision, and the rod or cone predominance increases over time, depending on whether ambient illumination levels increase or decrease. Changes to. As the eye adapts to a brighter environment, more cones are activated compared to the rod, while as the eye adapts to a darker environment, more rods compare to the cone. And is activated.

Scotopic vision typically occurs under light conditions where the illumination level is less than that for photopic vision. For example, scotopic vision occurs at illumination levels of about 10 ⁻² cd / m ² or less, including about 10 ⁻³ to 10 ⁻⁶ cd / m ² or about 10 ⁻³ cd / m ² or less. obtain. The rods are mainly active in scotopic vision. It should be understood that the illumination levels described herein for photopic, mesopic, and scotopic vision are examples. In some embodiments, the lighting level associated with each of the types of vision is based on user preferences and / or groups to which the user belongs (eg, based on gender, age, ethnicity, presence of visual abnormalities, etc.). May be arbitrarily assigned based on customizations.

  In some embodiments, the type of vision that is active in the user (photopic, mesopic, or scotopic) may be determined based on measurements of ambient lighting levels. For example, the display system may be configured to use a photosensor, such as an outward facing camera 112 (FIG. 9D), to measure ambient lighting levels. In some embodiments, the display system may be in communication with another sensor or device, which provides information about ambient lighting levels.

  Head mounted display systems may block or attenuate some of the ambient light so that the outward facing camera may not provide a brightness level that accurately reflects the amount of light impinging on the eye. Please understand that. In addition, the display system is also a light source that can project light onto the eye and modify the illumination level to which the eye is exposed when providing virtual content. In some other embodiments, an inward facing camera may be utilized to determine the brightness level. For example, the brightness level is roughly correlated with the size of the pupil. FIG. 17 diagrammatically illustrates an example of the relationship between pupil size and the amount of light incident on the user's eye. The x-axis shows values for luminance and the y-axis shows values for the pupil area. As a result, the display system may be configured to determine the user's pupil area and then extrapolate the brightness based on the present pupil area. For example, the display system may use an inward facing camera 500 (FIG. 6) to capture an image of the user's eye 210 and then analyze the image to determine the pupil area or other metric indicating the pupil area ( For example, it may be configured to determine the pupil diameter or width). For example, the area occupied by the pupil of the eye 210 in the image captured by the camera may be determined and then corrected for any scaling factors caused by the camera optics. Advantageously, using the pupil area, the step of determining the brightness level comprises blocking some ambient light, a reduction in the ambient brightness level caused by the display and also to the brightness level due to the light output of the display itself. Both contributions can be effectively considered.

With continued reference to FIG. 17, the display system is configured to determine whether the user's eyes are in photopic, mesopic, or scotopic mode based on the determined pupil area. May be. For example, the display system may reside in memory with a table or other stored information that defines the expected visual mode for a particular pupil area. As an example, consistent with the graph shown in FIG. 17, the display system exhibits photopic vision of a pupil area of about 3 mm ² or less, with a pupil area of up to about 38 mm ² of 3 mm ² or more. Pupillary areas larger than 38 mm ² that show mesopic vision may be categorized as showing scotopic vision. It should be appreciated that these brightness values and associated visual modes are examples and other values may be substituted. For example, different values may be applied to different users in response to input from the user, or different values may result in particular categories (eg, gender, age, ethnicity, visual abnormalities) in which the user may fall. It may be applied based on existence). In addition, it should be appreciated that the display system does not necessarily identify the particular visual mode. Rather, the display system may simply be configured to associate a particular measured pupil area with a particular resolution level or adjustment.

  In some embodiments, inputs from both the inward facing camera 510 (FIG. 6) and the outward facing camera 112 (FIG. 9D) may be utilized to determine the brightness level. For example, the display system may be configured to take an average (including a weighted average) of brightness levels determined using cameras 510 and 112. As mentioned above, the brightness level determined using the camera 510 is based on the imaging of the user's eye using the camera 510 to deviate from the size of the pupil area of the user's eye. May be inserted.

  It should be appreciated that rods and cones have different levels of visual acuity and different sensitivities to color and contrast. As a result, there is a difference in visual acuity and sensitivity to color and contrast at different ambient brightness levels, as the ambient brightness level affects whether the rods and / or cones are active. Advantageously, light level differences in visual acuity and sensitivity to color and contrast may be applied to provide an additional basis for reducing resolution, which is described above (e.g., 12A and 14) may be utilized in conjunction with fixation-based resolution changes, or may be utilized separately without specific fixation-based resolution changes.

  Referring now to FIG. 18, a schematic diagram of an example of a process 1800 for adjusting virtual content based on the amount of light incident on a user's eye is shown. For convenience, the process is a display system (eg, wearable display system 60 (FIG. 9D), which may include processing hardware and software, and, optionally, information to one or more computers or other processing units. May be provided to an external system, for example, processing may be offloaded to the external system, and information may be received from the external system).

  At block 1810, the display system determines the amount of light reaching the retina. Preferably, the decision is an estimate of the amount of light reaching the retina, rather than a direct measurement of light impinging on the retina. This estimation may be performed using the disclosed method for determining the brightness level, as discussed herein. For example, the brightness level may be assumed to correspond to the amount of light reaching the retina. As a result, the step of determining the amount of light reaching the retina is the step of determining the size of the user's pupil and / or a sensor configured to detect light such as an outward facing camera on the display device. May be used to determine the ambient brightness level.

  At block 1820, the display system adjusts the resolution of the virtual content to be presented to the user based on the amount of light found to reach the retina at block 1810. In some embodiments, adjusting the resolution of the virtual content comprises adjusting one or more of spatial resolution, color depth, and light intensity resolution of the virtual content. It is to be understood that the human visual system has maximum visual and spatial resolution, color, and light intensity sensitivity under photopic illumination levels. The ability to perceive differences in spatial resolution, color, and light intensity is diminished under mesopic illumination levels and further diminished under scotopic illumination levels.

  As a result, in some embodiments, a virtual object may be rendered with full or high spatial resolution if the amount of light present corresponds to a level for photopic vision (mesopic or (Compare spatial resolution, which will be used for scotopic vision). If the amount of light present is found to correspond to the level for mesopic vision level, the virtual object was reduced compared to the spatial resolution utilized for the virtual object under the photopic illumination level. It may be rendered with spatial resolution. If the amount of light is found to correspond to the scotopic level, the virtual object may be rendered with a lower spatial resolution than that used under mesopic or photopic illumination levels. The spatial resolution may be adjusted, for example, by reducing the number of polygons, etc., as described herein.

  The color depth or bit depth may likewise be adjusted according to the lighting level, the highest color depth used under photopic lighting level, the intermediate color depth used under mesopic lighting level, the lowest color Depth is used under scotopic illumination levels. It should be appreciated that the color depth may be adjusted by varying the number of bits used per color component of the pixel, with fewer bits comparable to lower color depth.

  Similarly, although not limited by theory, it is believed that the gradation in light intensity is greater as the illumination level progresses from photopic to mesopic to scotopic illumination levels. In other words, it is believed that the human visual system is able to discriminate smaller differences in light intensity as ambient lighting levels decrease. In some embodiments, the display system may be configured to display fewer gradations in light intensity as the illumination level progresses from photopic to mesopic, scotopic illumination levels. .. As a result, the maximum number of tones in the light intensity level is presented under the photopic illumination level, less tones are presented under the mesopic illumination level, and even less tones are in the scotopic illumination. Presented below the level.

  Additionally, in some embodiments, the display system may be capable of providing gray levels at more light intensities than are perceivable by the user. An example of this is illustrated in Figures 22a-22c and discussed further below. For example, a display system may be able to display 256 different levels of intensity for a given image pixel, but a user may only be able to perceive a smaller number of levels, for example 64 levels. In this instance, multiple possible light intensity levels are subsumed within a single one of the perceptible light intensity levels. For example, the display system may be capable of displaying four different light intensity levels, but the user may perceive all four as similar. In such a situation, where multiple possible light intensities are perceived by the user to be the same, in such a situation, the display system may choose from these values perceived to be similar to the lowest It may be configured to select the intensity value. As a result, the display system may be able to utilize lower intensity, thereby reducing the amount of power used to illuminate the display and achieving the desired light intensity. This may have particular advantages in display systems where the individual pixels of the spatial light modulator are themselves light emitters such as organic and inorganic LEDs. In some embodiments, the number of gray levels decreases with decreasing ambient lighting levels, and the display system groups together a larger number of possible light intensity levels, with a group of lowest light intensities. Configured to display.

  With respect to the virtual content to be displayed, one, two, or all three of spatial resolution, color depth, and light intensity resolution are directed to the user in light conditions (light reaching the user's retina. It should be understood that it may be varied based on the amount of). These adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions, as disclosed herein, make adjustments to resolution based on distance from the fixation point of the user's eye. Instead, it may be performed on the entire virtual content. In some other embodiments, adjustments to spatial resolution, color depth, and / or light intensity resolution based on light conditions may be made in conjunction with adjustments to resolution based on distance from a fixation point (eg, See Figures 12A and 14). In some embodiments, if the resolution decreases with distance from the fixation point, the profile of the decrease on a given plane (on the x and y-axes) preferably corresponds to the corresponding portion of the retina. The profile of crossing cone density changes is fitted.

  In some embodiments, adjustments to spatial resolution, color depth, and / or light intensity resolution, as described herein, are preferably modes of vision that are active at a given time (photopic vision). , Mesopic, or scotopic). These adjustments may change dynamically as the mode of vision changes. For example, as the user progresses from photopic to scotopic vision, resolution may decrease, as discussed herein. Conversely, as the user progresses from scotopic to mesopic vision, the resolution of the virtual content may increase. It should be appreciated that coupling resolution adjustments to a particular mode of vision does not require the user to make a specific determination to be in that particular mode. Rather, the display system may simply be configured to associate a particular ambient illumination level or range of pupil sizes with a particular resolution regardless of spatial resolution, color depth, or light intensity resolution. In addition, the resolution adjustment is preferably tied to three levels of light conditions (corresponding to the three modes of vision) as discussed herein, although in some embodiments the resolution adjustment is May be tied to two levels of light conditions or more than three levels of light conditions.

  Also, the resolution adjustment may occur in real time (eg, as the ambient light conditions change), or may be delayed for a set duration to allow the human visual system to see before the resolution adjustment is made to the virtual content. It should be appreciated that it may be possible to accommodate existing light conditions. Without being limited by theory, it is believed that the human visual system requires a certain period of time to adapt to different lighting levels, which increases as the lighting level decreases. As a result, in some embodiments, the adjustment in resolution due to changing the illumination level is exposed to the particular illumination level for a user-set amount of time (eg, substantially persistently). Not exposed until exposed). For example, the set amount of time may be 5 minutes, 10 minutes, 15 minutes, or 20 minutes.

  With continued reference to FIG. 18, at block 1830, virtual content is presented to the user. Presentation of the virtual content may occur as discussed herein, eg, as in block 1208 of FIG. 12A or block 1412 of FIG.

  Referring now to FIG. 19, an example of a change in resolution detectable by the user's eye as the amount of light incident on the eye changes is schematically illustrated. This figure illustrates an example of the sensitivity of the human visual system to spatial resolution under different visual modes. Scotopic vision occurs in the low light region 1910, mesopic vision occurs in the mid-light region 1920, and photopic vision occurs in the bright light region 1930. As shown, the sensitivity to spatial resolution decreases substantially as the ambient illumination level decreases. In some embodiments, the adjustments to spatial resolution discussed above with respect to FIG. 18 correspond to the contours of the curves shown. For example, for a given light level in photopic or scotopic mode, virtual content is rendered with sufficient spatial resolution to meet or exceed the resolution values shown on the y-axis.

  Referring now to FIG. 20, it should be appreciated that different photoreceptors may be used to perceive light of different wavelengths or colors. FIG. 20 schematically illustrates an example of the difference in the sensitivity of the eye to different colors of light at different levels of illumination. The difference in duration on the x-axis is a reflection of the amount of time the human visual system typically requires to adapt to a particular ambient lighting level so that a particular mode of vision is activated. is there. It should be noted that at ambient lighting levels corresponding to scotopic and mesopic vision, photoreceptors for red light may no longer be active, while photoreceptors for blue light may Active under minimal light conditions. It should be appreciated that red, green, and blue light correspond to the colors most typically used as primary colors in display systems to form full-color images (e.g., herein with respect to Figures 8-9B. As discussed). In some embodiments, the display system may be configured to vary the rendering of different color images depending on the ambient lighting level.

  Referring now to FIG. 21, a schematic diagram of an example of a process 2100 for adjusting virtual content formed using multiple primary color images is shown, wherein the resolution adjustment is based on the colors of the primary color images. Be seen. At block 2110, the display system uses the plurality of primary color images to provide virtual content to be presented. These may be different images of different primary colors that will be directed to different waveguides, as discussed with respect to Figures 8-9B. As a result, in some embodiments, each stream of images of different primary colors may be rendered separately. The step of providing virtual content to be presented using multiple primary color images may include utilizing a display system to output image streams of different primary colors to form a full color image.

  At block 2120, the display system may adjust the resolution of the primary color image based on the color. For example, the display system may select a color image of one of these primary colors for resolution adjustment. For example, the selection may be based on a lighting level determination, as described above with respect to block 1810 of FIG. As shown in FIG. 19, some primary colors may not be perceived by the user at some lighting levels. The display system may have stored therein information about the illumination levels and the primary colors invisible at those levels. If there is a match between the illumination levels and the invisible primaries at those levels, the image of that primaries may be selected for adjustment. In some circumstances, one adjustment may simply be to not render or display the primary color image if the ambient lighting level is such that the user is not expected to perceive the color. .. For example, under scotopic illumination levels, the display system may be configured to render or not display the primary color red image.

  With continued reference to FIG. 21, at block 2130, virtual content is presented to the user. Presentation of virtual content may occur as discussed herein, for example, as in block 1208 of FIG. 12A or block 1412 of FIG.

22A-22C, as discussed above, and without being limited by theory, the human visual system's ability to perceive gray scales in light intensity varies with ambient lighting level. It is thought that. 22A-22C show examples of contrast sensitivity that change as the amount of light incident on a user's eye decreases. For example, FIG. 22A may be understood to show contrast sensitivity under photopic conditions, FIG. 22B may be understood to show contrast sensitivity under mesopic conditions, and FIG. 22C shows scotopic conditions. It can be understood to indicate the contrast sensitivity below. FIG. 22A shows a progression 2100 from grayscale 2110 ₁ to 2110 _i , going from high light intensity at the top to low light intensity at the bottom. Similarly, FIG. 22B shows progression 2102 from grayscale 2110 ₁ to 2110 _i , going from high light intensity to low light intensity. Similarly, FIG. 22C shows a progression 2104 from grayscale 2110 ₁ to 2110 _i , going from high light intensity to low light intensity. Boxes 2120, 2130, 2140 indicate groups of intensity gradations that are perceived as identical by the user. The size of these groups is expected to increase with decreasing ambient lighting levels, as shown. As a result, as described above with respect to FIG. 18, in some embodiments, the display system may use the lowest intensity value within each group (eg, within each of the boxes 2120, 2130, 2140). It may be configured.

Referring now to FIG. 23, an example of a representation of the optic nerve and peripheral blind spots of the user's eye is illustrated. In some embodiments, in addition to, or as an alternative to, any of the resolution adjustments disclosed herein, the display system displays the content in various locations where the content is not expected to be perceivable by the user. It may be configured to refrain from rendering. FIG. 23 illustrates left and right eyes 210 _L and 210 _R , respectively. Each eye has a separate optical axis 1003A and 1003B and optic nerves 2300 _L and 2300 _R. Each optic nerve 2300L and 2300 _R is, its point of contact to the individual eye 210 _L and 210 _R are blind spots are present. These blind spots prevent the viewer from seeing the content in the direction of rays 2302 _L and 2302 _R. In addition, there is a region at the periphery of each eye where the content cannot be seen by the opposite eye. For example, content in the left peripheral region P _L may be seen by the left eye 210 _L but not by the right eye 210 _R. On the other hand, the content of the right peripheral region _{P R} is can be seen by the right eye 210 _R, not seen by the left eye 210 _L. As a result, in some embodiments, the display system skips rendering content that would be mapped to the blind spots of each eye 210 _L and 210 _R , eg, content that falls on rays 2302 _L and 2302 _R. May be configured to do so. As addition or alternatively, in some embodiments, the display system, if the content falls right peripheral region P _L, be configured to omit rendering the content on the left-eye 210 _L Well and / or the display system may be configured to skip rendering the content to the right eye 210 _R if the content falls within the left peripheral region P _L. The location of blind spots and / or surrounding areas may be preset, for example, based on an average of the user's population, and / or test and virtual objects using content displayed in various locations are visible. It should be appreciated that the input from the user indicating whether or not may be adjusted and calibrated for the particular user.

III. Multiple image streams to provide content with different resolutions

  In some embodiments, the foveated images having high and low spatial resolution regions each have two or more spatially overlapping image streams, each having a different resolution (eg, different perceived pixel density). May be formed by. For example, one of the image streams, eg, a low resolution image stream, may form an image having a wide field of view, and another of the image streams, eg, a high resolution image stream, an image having a narrow field of view. May be formed. Narrow-field images and wide-field images may contain similar content, but are viewed by the user at different resolutions or pixel densities. These images may be overlaid on top of each other (eg, occupy the same place in space at the same time or in close proximity in time so that the viewer perceives the images to be presented simultaneously). Thus, a viewer may receive an aggregate image with high resolution in a constrained portion of its field of view and a low resolution image over a larger portion of its field of view. Preferably, as discussed herein, the high resolution portion is mapped to the foveal vision area of the user's eye while the low resolution portion is mapped to the peripheral vision area of the user's eye. Therefore, the difference in resolution between the high and low resolution portions of the image is preferably not easily perceptible to the user.

  In some environments, display systems for displaying high and low resolution images utilize the same spatial light modulator to form both images. Therefore, spatial light modulators have pixels of fixed size and density. In display systems with fixed size and density pixels, the increase in angular field of view (FOV) occurs at the expense of spatial or angular resolution, eg, as dictated by the Lagrange invariant. For example, if an SLM with a fixed number of pixels is used to form both high and low resolution images, diffusing those pixels across the entire field of view means that they are small in the total field of view. Constraining parts will provide an image with a lower apparent resolution. That is, the pixel density of the high resolution image is higher than the pixel density of the low resolution image. As a result, there is generally an inverse correlation between FOV and angular resolution. Since FOV and angular resolution affect image visibility and quality, this tradeoff imposes constraints on user experience and final achievable FOV and angular resolution in AR or VR systems. As will be apparent from the discussion herein, in some embodiments the term "resolution" may be used to refer to "angular resolution".

  Head-mounted or wearable display devices can be configured to provide an immersive user experience by projecting virtual content directly into the user's eyes. Providing a wide FOV image uniformly with high resolution across the FOV can be beneficial, but the physiological limitations of the human visual system are that the high resolution image that the user positions within the peripheral region of the user's field of view. Can prevent you from viewing or even recognizing. The imperceptibility of high-resolution images in this peripheral region is caused by the characteristics of the human eye retina, which contains two types of photoreceptors, rod cells and cone cells. The cone is more involved in sensitive (detailed) vision. The rods and cones are differently distributed within the human eye. The highest concentration of pyramidal cells is found in the fovea (ie, the center of the retina), while the highest concentration of rod cells is found in the area directly surrounding the fovea (ie, the periphery of the retina). Due to this non-uniform distribution of rod and cone cells, the fovea is involved in sharp central vision (also called foveal vision). Visual acuity decreases with increasing distance from the fovea.

  For AR or VR applications, the headset is typically worn by one user at a time. The headset limits the display of high-resolution content to the area within the wide field of view that is currently in focus by the user, thereby eliminating the user's inability to perceive all the details of the wide field stream of the image at one time. Can be configured to utilize. In this way, the headset allows the user to see a high resolution wide FOV stream of images without the need for processing power, which would otherwise be required to produce high resolution content across the entire field of view. Can be provided. The stream of images presented to the user can take many forms and will generally be referred to as an image stream. For example, the image stream can show static images by persistently displaying the same image to the user, or it can show motion by displaying a different stream of images. In some embodiments, the headset can be configured to display more than one image stream simultaneously, different image streams can have different angular resolutions, and different regions of the user's FOV. Can extend across. The image stream associated with the AR system is designed such that the AR system is designed to mix virtual content with real-world content, so that the content cannot be displayed generally across the particular area to which it is assigned. Please note.

  According to some embodiments, the first image stream and the second image stream may be presented to the user at the same time or in rapid succession such that the two image streams appear to be displayed simultaneously. it can. The first image stream can have a wide FOV and a low resolution that can encompass the user's vision and evoke an immersive experience for the user. The portion of the first image stream that corresponds to the instantaneous portion of the FOV covered by the second image stream may be turned off in some embodiments. The second image stream may be dynamically displayed within the boundaries of the first image stream according to the user's current fixation point as determined in real time using eye gaze tracking techniques, a narrow FOV and high resolution. Can have. In other words, the second image stream can be shifted as the eye gaze of the user changes so that the second image stream permanently covers the fovea of the user. In some embodiments, the first image stream is presented to the user at a fixed location as the second image stream is offset with respect to the first image stream. In some other embodiments, both the first image stream and the second image stream are shifted according to the user's current fixation point.

  The content of the second image stream may include a subset of the content of the first image stream with a higher resolution than the first image stream, overlaid on the first image stream and appropriately for it. Can be aligned. Since the higher resolution second image stream overlays a portion of the first image stream within the fovea of the user, the modulation transfer function (MTF) in the area containing the higher resolution image stream is increased. It In some embodiments, the subset of content of the first image stream overlaid by the second image stream may be turned off or presented at a lower intensity. In this way, the user can perceive the combination of the first image stream and the second image stream to have both wide FOV and high resolution. Such a display system can offer several advantages. For example, a display system may have a relatively small form factor, while saving a computing resource and computing power, while providing a better user experience.

  In an embodiment, the light intensity in the boundary region of the first image stream and the second image stream tapers to a value below the intended image brightness, the first image stream and the second image stream. The border areas of are overlapped. In the overlapping area, the sum of the light intensities due to the two image streams can be relatively constant and equal to the intended image brightness. Upon traversing the overlap region from the first image stream side to the second image stream side, the MTF is changed from a first value equal to or closer to the MTF of the first image stream to an MTF of the second image stream. Change to a second value that is equal or closer. In this way, it is possible to avoid creating sharp boundaries between the regions served by the two image streams, which in some situations may be perceptible to the user.

  According to some embodiments, the first light beam associated with the first image stream and the second light beam associated with the second image stream are combined into a combined light beam using a multiplexing method. Can be multiplexed into the beam. For example, time division multiplexing, polarization division multiplexing, wavelength division multiplexing, and the like can be used, according to various embodiments. The combined light beam can be directed to one or more optical elements that serve to demultiplex the combined light beam into two separate optical paths. For example, a beam splitter or optical switching element such as a polarizing beam splitter (PBS) or a dichroic beam splitter can be used to split the combined light beam, depending on the method of multiplexing used. Once separated, the first light beam associated with the first image stream and the second light beam associated with the second image stream are routed through their separate optical paths and ultimately , Can be provided to the user as output.

  According to some embodiments, the first light beam associated with the first image stream causes the first image stream to be presented with a wider FOV and lower angular resolution (dependent on the invariant. As described above), the second light beam associated with the second image stream may be angularly expanded by an optical element in the first optical path, while the second light beam is not expanded angularly. It is expanded or expanded by an amount less than the amount of expansion applied to the first light beam associated with the first image stream. In this way, the second image stream can be presented with a narrower FOV and higher angular resolution than the first image stream (as influenced by the invariant).

  FIG. 24 shows a field schematic depicting the outer perimeter of an exemplary monocular field of view 3002 for a human eye in a two-dimensional angular space. As shown in FIG. 24, the temple-nose and inferior-upper axes of the field-of-view schematic serve to define a two-dimensional angular space into which the outer perimeter of monocular field of view 3002 is mapped. Thus, the view schematic of FIG. 24 may be considered equivalent to or similar to the “Goldmann” view map or plot for the human eye. The field schematic shown in FIG. 24 represents the field schematic for the human left eye, as shown by the depicted arrangement of the temple-nose and inferior-upper axis. The field of view may vary slightly from person to person, but the depicted field of view approximates what many humans can see with their left eye. A field schematic, depicting the outer perimeter of the exemplary monocular field of view of the right eye, is of the version of the field schematic of FIG. 24 with the temple-nasal axis and outer perimeter of the monocular field of view 3002 mirrored about the lower-upper axis. It can be similar to one.

  The field schematic of FIG. 24 further depicts the outer perimeter of an exemplary moving eye field of view 3004 for the human eye, which represents a portion of the monocular field of view 30022 in angular space that a person may fix. In addition, the view schematic of FIG. 24 also depicts the perimeter of an exemplary foveal field 3006 for the human eye, which is in angular space within a direct view of the fovea of the human eye at a given time. A part of the monocular visual field 3002 is shown. As depicted, a person's foveal area 3006 can move anywhere within the ocular visual field 3004. The portion of monocular field of view 3002 outside central fovea 3006 in angular space may be referred to herein as the peripheral region of the human field of view. Displaying reduced resolution images outside the fovea 3006 may be perceptible because the human eye's ability to distinguish high levels of detail outside the fovea 3006 is very limited. Low, it may allow substantial savings in power consumption for processing components involved in generating content for a display.

  FIG. 25A illustrates an exemplary wearable display device 4050 configured to provide virtual content to a user, according to some embodiments. Wearable display device 4050 includes a main display 4052 supported by a frame 4054. The frame 4054 can be attached to the user's head using attachment members in the form of temple arms 4006.

  25B, an exemplary embodiment of an AR system configured to provide virtual content to a user will now be described. In some embodiments, the AR system of FIG. 25B may represent the system to which the wearable display device 4050 of FIG. 25A belongs. The AR system of FIG. 25B uses a stacked light guiding optics element assembly 4000 and generally includes an image generation processor 4010, a light source 4020, a controller 4030, and a spatial light modulator (“SLM”) 4040. It includes an optical system 4060 and at least one set of stacked eyepiece layers or light guiding optics elements (“LOE”, eg, planar waveguide) 4000 that act as a plurality of planar focusing systems. The system may also include an eye tracking subsystem 4070. It should be appreciated that other embodiments may have multiple sets of stacked LOE 4000s, but the following disclosure will focus on the exemplary embodiment of FIG. 25B.

  The image generation processor 4010 is configured to generate virtual content that will be displayed to the user. The image generation processor may convert the image or video associated with the virtual content into a format that can be projected to the user in 3-D. For example, when generating 3-D content, the virtual content needs to be formatted such that certain portions of the image are displayed in certain depth planes while others are displayed in other depth planes. Can be. In one embodiment, the images may all be generated in a particular depth plane. In another embodiment, the image generation processor may be programmed to provide slightly different images to the right and left eyes 210 so that the virtual content, when viewed together, appears coherently and comfortably to the user's eyes. Good.

  Image generation processor 4010 may further include memory 4012, GPU 4014, CPU 4016, and other circuitry for image generation and processing. The image generation processor 4010 may be programmed with the desired virtual content to be presented to the user of the AR system of Figure 25B. It should be appreciated that in some embodiments, the image generation processor 4010 may be stored within a wearable AR system. In other embodiments, the image generation processor 4010 and other circuitry may be stored in a belt pack, which is coupled to wearable optics. The image generation processor 4010 is operably coupled to a light source 4020 and one or more spatial light modulators (described below) that project light associated with the desired virtual content.

  The light source 4020 is compact and has high resolution. The light source 4020 includes a plurality of spatially separated sub-light sources 4022 operably coupled to a controller 4030 (described below). For example, light source 4020 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, light source 4020 may include LEDs or lasers of similar color, one each coupled to a specific area of the display's field of view. In another embodiment, the light source 4020 may comprise a wide area emitter, such as an incandescent or fluorescent lamp, with a mask overlaying for segmentation of the emission area and location. The sub-light source 4022 is directly connected to the AR system of FIG. 2B in FIG. 2B, but as long as the distal ends of the optical fibers (away from the sub-light source 4022) are spatially separated from each other. , May be connected to the system via an optical fiber (not shown). The system may also include a collector (not shown) configured to collimate the light from light source 4020.

  The SLM 4040 may be reflective (eg, DLP DMD, MEMS mirror system, LCOS, or FLCOS), transmissive (eg, LCD), or emissive (eg, FSD or OLED) in various exemplary embodiments. Good. The type of spatial light modulator (eg, speed, size, etc.) can be selected to improve the creation of 3-D perception. While DLP DMDs operating at higher refresh rates can be easily incorporated into stationary AR systems, wearable AR systems typically use smaller size and power DLPs. The power of the DLP changes the way the 3-D depth plane / focal plane is created. The image generation processor 4010 is operably coupled to the SLM 4040, which encodes the light from the light source 4020 with the desired virtual content. Light from light source 4020 may be encoded with image information as it reflects from, exits from, or passes through SLM 4040.

  Referring back to FIG. 25B, the AR system also includes input optics configured to direct light from light source 4020 (ie, a plurality of spatially separated sub-light sources 4022) and SLM 4040 to LOE assembly 4000. Includes system 4060. Input optics system 4060 may include one or more lenses configured to direct light into LOE assembly 4000. The injection optics system 4060 is spatially separated from the sub-light source 4022 of the light source 4020 and is configured to form a spatially separated and distinct pupil adjacent to the LOE 4000, corresponding to distinctly different beams ( To the individual focus points of the beam exiting the injection optics system 4060). Input optics system 4060 is configured such that the pupils are spatially displaced from each other. In some embodiments, the input optics system 4060 is configured to spatially displace the beam only in the X and Y directions. In such an embodiment, the pupil is formed in one X, Y plane. In other embodiments, input optics system 4060 is configured to spatially displace the beam in the X, Y, and Z directions.

  The spatial separation of the light beams forms distinctly different beams and pupils, which means that each incoupling grating is primarily addressed (eg, intersected) by only one distinctly distinct beam (or group of beams). Or impingement), allowing the installation of internally coupled gratings in distinctly different beam paths. This, in turn, facilitates the incidence of spatially separated light beams into the individual LOE 4000s of LOE assembly 4000 while the incidence of other light beams from other sub-sources 4022 of the plurality (ie, Crosstalk) to a minimum. A light beam from a particular sub-light source 4022 is incident on an individual LOE 4000 through an internal coupling grating (not shown in FIG. 25B, see FIGS. 24-26) above it. The individual LOE4000 internal coupling gratings are spatially separated from the plurality of sub-light sources 4022 such that each spatially separated light beam intersects only one LOE4000 internal coupling grating. Configured to interact. Therefore, each spatially separated light beam is mainly incident on one LOE4000. Thus, the image data encoded by the SLM 4040 onto the light beam from each of the sub-light sources 4022 can be effectively propagated along a single LOE 4000 for delivery to the user's eye 210.

  Each LOE 4000 is then configured to project an image or sub-image that appears to arise from the desired depth plane or FOV angular position onto the user's retina. The individual LOE 4000s and sub-light sources 4022 thus selectively project images (synchronously encoded by the SLM 4040 under the control of the controller 4030) that appear to arise from different depth planes or positions in space. can do. By sequentially projecting the image at each of a plurality of individual LOE 4000s and sub-light sources 4022 at a sufficiently high frame rate (eg, 360Hz for 6 depth planes at an effective full volume frame rate of 60Hz). The 25B system can generate 3-D images of virtual objects in various depth planes that appear to be present simultaneously in the 3-D image.

  The controller 4030 is in communication with and operably coupled to the image generation processor 4010, the light source 4020 (sub-light source 4022), and the SLM 4040 to direct the SLM 4040 the light beam from the sub-light source 4022 to the appropriate image from the image generation processor 4010. Coordinate the synchronized display of images by instructing them to encode with information.

  The AR system also includes an optional eye tracking subsystem 4070 configured to track the user's eye 4002 and determine the user's focus. In one embodiment, only a subset of sub-light sources 4022 may be activated and illuminate a subset of LOE 4000s based on input from the eye tracking subsystem, as discussed below. Based on input from the eye tracking subsystem 4070, one or more sub-light sources 4022 corresponding to a particular LOE 4000 are active so that an image is generated in the desired depth plane that matches the user's focus / accommodation. It may be embodied. For example, if the user's eyes 210 are parallel to each other, the AR system of FIG. 25B is configured to deliver collimated light to the user's eyes so that the image appears to arise from optical infinity. , LOE4000 corresponding sub-light source 4022 may be activated. In another example, if the eye tracking subsystem 4070 determines that the user's focus is 1 meter away, then there is a sub-light source 4022 corresponding to the LOE4000 that is configured to focus approximately within that range. Alternatively, it may be activated. It is understood that in this particular embodiment, only one group of sub-light sources 4022 is activated at any given time, while the other sub-light sources 4020 are deactivated, saving power. I want to be done.

  FIG. 25C schematically illustrates a light path within an exemplary viewing optics assembly (VOA) that may be used to present a digital or virtual image to a viewer, according to some embodiments. In some embodiments, VOAs may be incorporated into a system similar to wearable display device 4050 as depicted in Figure 25A. The VOA includes a projector 4001 and an eyepiece 200 that may be worn around the eyes of a viewer. Eyepiece 4000 may, for example, correspond to LOE4000 as described above with reference to Figure 25B. In some embodiments, projector 4001 may include a group of red LEDs, a group of green LEDs, and a group of blue LEDs. For example, projector 201 may include two red LEDs, two green LEDs, and two blue LEDs, according to some embodiments. In some implementations, the projector 4001 and its components (eg, LED light source, reflective collimator, LCoS SLM, and projector relay) as depicted in FIG. 25C are described above with reference to FIG. 25B. Optional light source 4020, sub-light source 4022, SLM 4040, and input optical system 4060 may represent or provide functionality of one or more of the following. Eyepiece 4000 may include one or more eyepiece layers, each of which may represent one of LOE 4000s as described above with reference to FIG. 25B. Each eyepiece layer of eyepiece 4000 may be configured to project an image or sub-image that appears to result from an individual desired depth plane or FOV angular position onto the retina of the viewer's eye.

  In one embodiment, the eyepiece 4000 includes three eyepiece layers, one for each of the three primary colors, red, green, and blue. For example, in this embodiment, each eyepiece layer of eyepiece 4000 may be configured to deliver collimated light to the eye, appearing to originate from an optical infinity depth plane (0 diopters). In another embodiment, the eyepiece 4000 includes six eyepiece layers, one set for each three primary colors configured to form a virtual image in one depth plane, and a virtual image. A separate set of eyepiece layers may be included for each of the three primary colors configured to form in different depth planes. For example, in this embodiment, each eyepiece layer within the set of eyepiece layers of eyepiece 4000 is designed to deliver collimated light to the eye, appearing to originate from the optical infinity depth plane (0 diopters). While each eyepiece layer in another set of eyepiece layers of eyepiece 4000 may be configured to a collimated light that appears to originate from a distance of 2 meters (0.5 diopters). It may be configured to deliver. In other embodiments, eyepiece 4000 may include more than two eyepiece layers for every three primary colors for more than two different depth planes. For example, in such embodiments, yet another set of eyepiece layers may each be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter).

  Each eyepiece layer comprises a planar waveguide and may include an internal coupling grating 4007, an orthogonal pupil expander (OPE) region 4008, and an exit pupil expander (EPE) region 4009. Further details regarding in-coupling gratings, orthogonal pupil expansion, and exit pupil expansion can be found in US patent application Ser. No. 14 / 555,585 and US patent application Ser. No. 14 / 726,424, the contents of which are As expressly incorporated herein by reference in its entirety). Still referring to FIG. 25C, projector 4001 projects image light onto incoupling grating 4007 in eyepiece layer 4000. The internal coupling grating 4007 couples the image light from the projector 4001 into the waveguide and propagates it in the direction toward the OPE region 4008. The waveguide propagates the image light in the horizontal direction by total internal reflection (TIR). The OPE region 4008 of the eyepiece layer 4000 also includes a diffractive element that combines a portion of the image light propagating in the waveguide and redirects it towards the EPE region 4009. More specifically, the collimated light propagates by the TIR horizontally along the waveguide (ie, relative to the view of FIG. 25C), thereby repeating the diffractive elements of the OPE region 4008. Cross. In some embodiments, the diffractive elements of OPE region 4008 have a relatively low diffraction efficiency. This means that at each point of intersection with the diffractive element of the OPE region 4008, a certain percentage (eg, 10%) of light is diffracted vertically downward toward the EPE region 4009, and a certain percentage of light is passed through the TIR. Continue on its original orbit horizontally along the waveguide. Thus, at each point of intersection of the OPE region 4008 with the diffractive element, additional light is diffracted downward toward the EPE region 4009. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light is extended horizontally by the diffractive elements of the OPE region 4008. The expanded light that is outcoupled from the OPE region 4008 enters the EPE region 4009.

  The EPE region 4009 of the eyepiece layer 4000 also includes diffractive elements that combine some of the image light propagating in the waveguide and redirect it towards the viewer's eye 210. Light incident on EPE region 4009 propagates vertically along the waveguide (ie, relative to the view of FIG. 25C) by TIR. At each point of intersection between the propagating light and the diffractive elements of the EPE region 4009, a percentage of the light is diffracted towards the adjacent face of the waveguide, causing the light to escape the TIR and Emerging from the surface of the eye and allowing it to propagate towards the viewer's eye 210. In this method, the image projected by the projector 4001 can be visually recognized by the eyes 210 of the viewer. In some embodiments, the diffractive elements of EPE region 4009 may be designed or configured to have a phase profile that is the sum of a linear grating and a radially symmetric diffractive lens. Radially symmetric lens sides of the diffractive elements of the EPE region 4009 additionally impart a focus level to the diffracted light, shaping the optical wavefront of the individual beams (eg, imparting curvature), and And steering at an angle that matches the designed focus level. Each beam of light that is outcoupled by the diffractive elements of the EPE region 4009 may extend geometrically to an individual focal point, located in front of the viewer, and centered on a radius of the individual focal point. With an accompanying convex wavefront profile, an image or virtual object may be produced at a given focal plane.

Descriptions of such viewing optics assemblies and other similar settings are further provided in US patent application Ser. No. 14 / 331,218, US patent application Ser. No. 15 / 146,296, and US patent application Ser. No. 14 / 555,585 ( All of which are incorporated herein by reference in their entirety). In some embodiments, an exemplary VOA includes one or more components described in any of the patent applications described above with reference to FIG. 25C and incorporated herein by reference. It may be included and / or may take the form thereof.

IV. High field of view and high resolution foveated display using multiple optical paths

  26A-26D illustrate exemplary rendering viewpoints that will be used and the light fields that will be produced in the AR system for every two exemplary eye orientations. In FIG. 26A, the viewer's eye 210 is oriented in a first manner with respect to the eyepiece 5000. In some embodiments, eyepiece 5000 may be similar to a stack of LOEs or eyepiece 4000, as described above with reference to FIGS. 25B and 25C. More specifically, in this example, the viewer's eye 210 is oriented so that the viewer may be able to see the eyepiece 5000 in a relatively straight-ahead direction. In some embodiments, the AR system, to which the eyepiece 5000 belongs, may be similar to the AR system as described above with reference to FIG. One or more actions for presenting virtual content may be performed on the plane at one or more distances in front of the viewer's eyes 210.

  The AR system may determine the viewpoint in the rendering space based on the position and orientation of the viewer's head, from which the viewer views 3-D virtual content in the rendering space, such as virtual objects. To do. As described in more detail below with reference to FIG. 29A, in some embodiments, such an AR system includes one or more sensors and collects data from these one or more sensors. May be utilized to determine the position and / or orientation of the viewer's head. The AR system may include one or more eye tracking components, such as one or more components of the eye tracking subsystem 4070 described above with reference to FIG. 25B, as well as one or more such sensors. Good. Using such data, the AR system effectively maps the position and orientation of the viewer's head in the real world to specific locations and specific angular positions within the 3D virtual environment. Creating a virtual camera positioned at a specific location within the 3D virtual environment and oriented at a specific angular position within the 3D virtual environment relative to a specific location within the 3D virtual environment, such as viewing as would be captured by the virtual camera Virtual content for a person may be rendered. For more details on discussing the real-world and virtual-world mapping process, see US Patent Application No. 15 / 296,869 entitled "SELECTING VIRUAL OBJECTS IN A THREE-DIMENSIONAL SPACE" (for all purposes, see Are hereby incorporated by reference in their entirety).

  In some embodiments, the AR system may provide one such head-tracking virtual camera as the viewer's eyes and / or orbits are physically separated from each other and thus consistently located at different locations. May be created or dynamically repositioned and / or reoriented for the viewer's left eye or orbit, and another such head-tracking virtual camera for the viewer's right eye or orbit. Virtual content rendered from the perspective of a head-tracking virtual camera associated with the viewer's left eye or orbit is an eyepiece on the left side of a wearable display device such as those described above with reference to FIGS. 25A-25C. Virtual content that may be presented to the viewer through the lens and rendered from the perspective of the head tracking virtual camera associated with the viewer's right eye or orbit is presented to the viewer through the eyepiece on the right side of the wearable display device. You will get it. Head-tracking virtual cameras may be created and / or dynamically repositioned for each eye or orbit based on information about the current position and orientation of the viewer's head. The position and orientation of the camera may be independent of either the individual eye sockets of the viewer or the position or orientation of each eye of the viewer relative to the viewer's head. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process can be found in US patent application Ser. Are hereby incorporated by reference in their entirety).

The AR system of FIG. 26A creates or dynamically repositions and / or reorients such a head-tracking virtual camera and renders virtual content from the head-tracking virtual camera's viewpoint (viewpoint 5010). Of light can be projected through the eyepiece 5000 onto the retina of the viewer's eye 210. As shown in FIG. 26A, head-tracking rendering viewpoint 5010 may provide a FOV diagonally, horizontally, and / or vertically spanning an area of ± θ ₃₁₀ angular units. As described in further detail below, in some embodiments the head-tracking rendering viewpoint 5010 may provide a relatively wide FOV. In such an embodiment, the AR system also creates or dynamically repositions and / or reorients another virtual camera for each eye or orbit that is different from and in addition to the head tracking virtual camera. Good. In the example of FIG. 26A, the AR system may render and present virtual content from the perspective of another virtual camera in the rendering space along with virtual content from the perspective of head tracking virtual camera 5010.

  For example, in such an embodiment, the AR system of FIG. 26A may create or dynamically reposition and / or reorient such a foveal tracking virtual camera based on the current line of sight of the viewer's eye 210. You may. As will be described in more detail below with reference to FIG. 29A, in some embodiments such an AR system is one of the eye tracking subsystems 4070 described above with reference to FIG. 25B. One or more eye tracking components, such as one or more components, to determine the current line of sight of the viewer, the current position and / or orientation of the viewer's eye 210 relative to the viewer's head, and the like. Good. Using such data, the AR system of FIG. 26A creates or dynamically repositions and / or reorients such a foveal tracking virtual camera, and virtual content is viewed from the foveal tracking virtual camera's viewpoint ( Light that is rendered from viewpoint 5020A) and that represents virtual content as rendered from viewpoint 5020A may be projected through eyepiece 5000 onto the fovea of viewer's eye 210.

As shown in FIG. 26A, foveal tracking rendering viewpoint 5020A may provide a narrower FOV than that of head tracking rendering viewpoint 5010. Thus, the FOV of fovea tracking rendering viewpoint 5020A can be seen to occupy the conical subspace of the FOV of head tracking rendering viewpoint 5010. That is, the FOV of fovea tracking rendering viewpoint 5020A may be a subfield of the FOV of head tracking rendering viewpoint 5010. For example, as shown in FIG. 26A, in the foveal tracking rendering viewpoint 320A, the relationship between the FOV of the head tracking rendering viewpoint 5010 and the foveal tracking rendering viewpoint 5020A is −θ ₃₁₀ ≦ −θ _320A ≦ θ _320A ≦ The FOV may be provided diagonally, horizontally, and / or vertically, as provided by θ ₃₁₀ , spanning an area of ± θ _{320 A} angular units. In some embodiments, the FOV of the head-tracking rendering viewpoint 5010 may be at least as wide as the visual field of the viewer's eye, which, in this embodiment, is given by the viewer's head. There will be a total conical space that the viewer's eye 210 can fix when held in position and orientation. Thus, in these examples, the head-tracking virtual camera and the foveal-tracking virtual camera may be positioned at substantially the same location in the rendering space, or both virtual cameras may be positioned at the position of the viewer's head. It may be located at a location in the rendering space that is a fixed distance from one another so that as the orientation and / or orientation changes, it can be translated linearly and / or angularly in the rendering space. For example, the head-tracking virtual camera may be located at a location in rendering space that corresponds to the center of rotation of the viewer's eye 210, while the fovea-tracking virtual camera provides a view between the center of rotation and the cornea. It may be located at a location in rendering space that corresponds to the area of the human eye 210. In fact, the Euclidean distance between two virtual cameras is similar, such that the Euclidean distance between two specific areas of the viewer's eye 210 or another rigid body may remain substantially constant at all times. , May remain substantially constant when translated in rendering space.

  The spatial relationship between each virtual camera within such a pair of virtual cameras may remain substantially fixed in the rendering space throughout the use of the AR system, however, in these embodiments, in the fovea The orientation of the tracking virtual camera may change with respect to the head tracking virtual camera as the viewer rotates his eye 210. Thus, the conical subspace of the FOV of the head tracking virtual camera occupied by the FOV of the foveal tracking virtual camera may change dynamically as the viewer rotates his or her eye 210.

  Further, virtual objects and other content falling within the foveal tracking rendering viewpoint 5020A can be rendered and presented at relatively high resolution by the AR system. More specifically, the virtual content in the FOV of the foveal tracking virtual camera is rendered and presented, and the resolution may be higher than the resolution in which the virtual content in the FOV of the head tracking virtual camera is rendered and presented. .. Thus, the highest resolution subfield of a given light field, which is externally coupled by the eyepiece 5000 and projected onto the retina of the viewer's eye 210, is that which reaches the fovea of the viewer's eye 210. possible.

  FIG. 3B shows that the viewer's eye 210 is externally coupled by the eyepiece 5000 while the viewer's eye 210 is oriented in a first manner as depicted in FIG. 26A and described above with reference to the viewer's eye 210. 5 illustrates an exemplary light field 5030A projected onto the retina of eye 210. Light field 5030A may include various angular light components that represent virtual content as would be captured in rendering space by the pair of virtual cameras described above. As described in more detail below with reference to Figures 26A et seq., A rendering space with a light and fovea tracking virtual camera representing virtual content as would be captured in the rendering space by the head tracking virtual camera. Light representing virtual content as would be captured in may be multiplexed by the AR system according to any of a variety of different multiplexing schemes. The adoption of such a multiplexing scheme may allow, in at least some instances, AR systems to operate with greater efficiency and / or occupy less physical space.

Still referring to FIG. 26B, it depicts virtual content as it would be captured in rendering space by a head-tracking virtual camera (eg, virtual objects and other content falling within head-tracking rendering viewpoint 5010). , Light field 5030A may include what is to be projected onto the retina of the viewer's eye 210 at an angle ranging from −θ ₃₁₀ to + θ ₃₁₀ angular units with respect to the viewer's eye 210. Similarly, a lightfield 5030A representing virtual content (eg, virtual objects and other content falling within the foveal tracking rendering viewpoint 5020A) as would be captured in rendering space by the foveal tracking virtual camera. Angular light components may include those that will be projected onto the retina of the viewer's eye 210 at angles ranging from −θ _320A to + θ _320A angular units relative to the viewer's eye 210. Such angular light components associated with foveal tracking rendering viewpoint 5020A occur within light field 5030A, and the spacing between the -θ _320A and + θ _320A angular units is the angular light component associated with head tracking rendering viewpoint 5010. Can occur in the light field 5030A and can be more regular than the spacing between the −θ ₃₁₀ and + θ ₃₁₀ angular units. Thus, the virtual content associated with the foveal tracking rendering viewpoint 5020A may be rendered and presented to the viewer, the resolution may be the virtual content associated with the head tracking rendering viewpoint 5010 rendered and presented to the viewer. , Can be higher than resolution.

In some embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 occurring within the light field 5030A is further at an angle relative to the viewer's eye 210 ranging from −θ _320A to + θ _320A angular units. It may include what will be projected onto the retina of the viewer's eye 210. In such an embodiment, such angular light components associated with head-tracking rendering viewpoint 5010 occur within light field 5030A such that the spacing between −θ _320A and + θ _320A angular units is foveal tracking rendering viewpoint 5020A. May be less regular than the spacing between the-[Theta] _320A and + [Theta] _320A angular units, where the angular light component associated with occurs in the light field 5030A. In another embodiment, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs within the light field 5030A is at an angle relative to the viewer's eye 210 at an angle ranging from −θ _320A to + θ _320A angular units. That would be projected onto the retina of the human eye 210. Thus, in these other embodiments, the angular light components associated with head-tracking rendering viewpoint 5010 that occur in light field 5030A are -θ ₃₁₀ to −θ _320A angular units of angles or θ _{320A to} θ ₃₁₀ . It may be what is to be projected onto the retina of the viewer's eye 210 at an angle.

  In FIG. 26C, the viewer's eye 210 is oriented with respect to the eyepiece 5000 in a second manner, which is different from the first manner in which the viewer's eye 210 is oriented with respect to the eyepiece 5000 in FIGS. 26A-26B. To be done. For purposes of example, the position and orientation of the viewer's head in FIGS. 26C-26D is the same as the position and orientation of the viewer's head as described above with reference to FIGS. 26A-26B. Can be treated as. Thus, FIGS. 26A-26B and 26C-26D may represent the viewer and AR system described above in the first and second time series stages, respectively. More specifically, in this example, the viewer's eye 210 has been rotated off center from a relatively straight orientation, as depicted in FIGS. 26A-26B.

  During the transition from the first stage to the second stage, the AR system of FIG. 26C does not change the viewer's head pose (eg, position and orientation), so refer to FIGS. 26A-26B. As explained above, for example, it may function to keep the head tracking virtual camera in the same position and orientation. Thus, in the second stage depicted in FIGS. 26C-26D, the AR system renders virtual content from the viewpoint of the head-tracking virtual camera (ie, head-tracking rendering viewpoint 5010), representing rendering of the virtual content. Light may be projected through the eyepiece 5000 onto the retina of the viewer's eye 210. Head tracking rendering viewpoint 5010 may remain static or relatively static throughout the first and second time series stages of FIGS. 26A-26D, but from the first stage to the second stage. During the transition, the AR system functions to adjust the orientation of the foveal tracking virtual camera in the rendering space based on the change in the line of sight of the viewer's eye 210 from the first stage to the second stage. You can That is, the AR system provides a foveal tracking rendering viewpoint 5020A such that a foveal tracking virtual camera as employed in the second stage provides a foveal tracking rendering viewpoint 5020C that is different from the foveal tracking rendering viewpoint 5020A. The foveal tracking virtual camera as employed in the first stage may be replaced or reoriented to provide. In the second stage, the AR system also renders the virtual content from the viewpoint of the foveal tracking virtual camera viewpoint 5020C and emits light representative of the rendering of the virtual content through the eyepiece 5000 onto the fovea of the viewer's eye 201. Can be projected onto.

26C-26D, the foveal tracking rendering viewpoint 5020C may occupy a conical subspace of the head tracking rendering viewpoint 5010 that differs from that of the foveal tracking rendering viewpoint 5020A. For example, as shown in FIG. 26C, the foveal tracking rendering viewpoint 5020C is displaced from the FOV of the foveal tracking rendering viewpoint 5020A by θ _320C angular units, diagonally, horizontally, and / or vertically ± θ _320A. An FOV can be provided that spans an area in angular units. That is, the foveal tracking rendering viewpoint 5020C may provide a FOV diagonally, horizontally, and / or vertically spanning an area of θ _320C ± θ _320A angular units.

FIG. 26D shows that the viewer's eye 201 is externally coupled by the eyepiece 5000 while the viewer's eye 201 is oriented in a second manner as depicted in FIG. 26C and described above with reference to it. 5 illustrates an exemplary light field 5030C projected onto the retina of eye 201. The light field 5030C may include various angular light components that represent virtual content as it would be captured in rendering space from the head-tracking rendering viewpoint 5010 and the foveal-tracking rendering viewpoint 5020C. The angular light component of the light field 5030C, which represents the virtual content as it would be captured in the rendering space from the head-tracking rendering viewpoint 5010, is -θ ₃₁₀ to + θ ₃₁₀ angular units to the viewer's eye 210. It may include what would be projected onto the retina of the viewer's eye 210 at a range of angles. However, as described above with reference to FIGS. 26A-26B, upon departure from the first phase, virtual content such as would be captured in the rendering space by the foveal tracking virtual camera (eg, The angular light component of the light field 5030C, which represents a virtual object and other content that falls within the foveal tracking rendering viewpoint 5020C, is θ _320C −θ _320A angular units to θ _320C + θ _320A relative to the viewer's eye 210. It may include what will be projected onto the retina of the viewer's eye 210 at angles spanning angular units.

The spacing between the θ _320C −θ _320A angular units and the θ _320C + θ _320A angular units at which such angular light components associated with the foveal tracking rendering viewpoint 320C occur within the light field 5030C is the head tracking rendering viewpoint. The angular light component associated with 5010 may be higher than the spacing between-[Theta] ₃₁₀ and + [Theta] ₃₁₀ angular units occurring in light field 5030C. Thus, the resolution with which virtual content associated with the foveal tracking rendering viewpoint 5020C can be rendered and presented to the viewer is noted that the resolution is -θ _320A to + θ _{320A with} respect to the viewer's eye 210. Virtual content associated with the head-tracking rendering viewpoint 5010 is rendered to the viewer, including virtual content represented by the angular light component that will be projected onto the retina of the viewer's eye 210 at unity angles. And can be presented and can be higher than the resolution.

In some embodiments, the angular light component associated with the head-tracking rendering viewpoint 5010 that occurs in the light field 5030C is further coupled to the viewer's eye 210 in terms of θ _320C −θ _320A angular units to θ _320C + θ _320A. It may include what will be projected onto the retina of the viewer's eye 210 at angles spanning angular units. In such an embodiment, the spacing between −θ _320C −θ _320A angular units and θ _320C + θ _320A angular units, such angular light components associated with head-tracking rendering viewpoint 310 occur within light field 5030C. , The angular light component associated with the foveal tracking rendering viewpoint 5020C occurs within the light field 5030C and may be less regular than the spacing between the θ _320C −θ _320A and θ _320C + θ _320A angular units. In another embodiment, the angular light component associated with the head-tracking rendering viewpoint 5010 occurring in the light field 5030C is θ _320C −θ _320A angular units to θ _320C + θ _320A angular units relative to the viewer's eye 210. What would be projected onto the retina of the viewer's eye 210 at angles ranging up to may be excluded. Thus, in these other embodiments, occur within the light field 5030C, angle light component associated with the head tracker rendering viewpoint 5010, -θ ₃₁₀ ~θ _320C -θ _320A angle unit angle and theta _320C + theta _320A The angle may be projected onto the retina of the viewer's eye 210 in angular units or angles of θ ~ ₃₁₀ angular units.

  26E-26F schematically illustrate exemplary configurations of images that may be presented to a user, according to some embodiments. Note that the grid squares in FIGS. 26E-26F graphically represent image points where fields of view 3002, 3004, and 3006 as described above with reference to FIG. 24 are defined in a two-dimensional angular space. I want to be done. A low resolution first image stream 5010E having a wide FOV can be displayed at a static location. A low resolution first image stream 5010E having a wide FOV displays one or more images of virtual content as would be captured by a first virtual camera having a static position and orientation in rendering space. Can be represented. For example, the low resolution first image stream 5010E is of virtual content as would be captured by a head tracking virtual camera, such as the head tracking virtual camera described above with reference to FIGS. 26A-26D. One or more images can be represented. The first image stream 5010E may include the user's vision and evoke an immersive experience for the user.

  A high resolution second image stream 5020E having a relatively narrow FOV can be displayed within the boundaries of the first image stream 5010E. In some embodiments, the second image stream 5020E is dynamically generated in real-time with respect to an angular position that matches the user's current fixation point based on data obtained using eye-tracking techniques. One or more images of the virtual content as they would be captured by a second different virtual camera with an orientation that is adjustable in rendering space may be represented. In these examples, the high resolution second image stream 5020E would be captured by a foveal tracking virtual camera, such as the foveal tracking virtual camera described above with reference to FIGS. 26A-26D. One or more images of different virtual content can be represented. In other words, the viewpoint in rendering space from which one or more images of the virtual content represented by the second image stream 5020E is captured is such that the viewpoint associated with the second image stream 5020E is centered on the user. It can be reoriented as the user's gaze changes so as to be permanently aligned with the fovea.

  For example, the second image stream 5020E includes virtual content located in the first region of the rendering space when the user's gaze is fixed in the first position, as illustrated in FIG. 26E. Can be included. As the user's line of sight moves to a second position that is different from the first position, the viewpoint associated with the second image stream 5020E can be adjusted so that the second image stream 5020E is adjusted. , And may include virtual content located in a second region of the rendering space, as illustrated in FIG. 26F. In some embodiments, the first image stream 5010E is a wide FOV, but has low angular resolution, as indicated by the sparse grid. The second image stream 5020E is a narrow FOV, but has high angular resolution, as shown by the dense grid.

  FIG. 26G schematically illustrates an exemplary arrangement of images that may be presented to a user, according to some other embodiments. As in FIGS. 26E-26F, the grid squares in FIG. 26G graphically represent image points defined in two-dimensional angular space. Similar to the configuration illustrated in FIGS. 26E-26F, a low resolution first image stream 5010G having a wide FOV contains virtual content as viewed from a head-tracking rendering perspective, while having a narrow FOV. The high resolution second image stream 5020G contains virtual content as viewed from the foveal tracking rendering viewpoint, which may be dynamically reoriented to match the user's current fixation point. Here, the perimeter of the FOV associated with the first image stream 5010G can form a rectangular boundary with rounded corners, and the perimeter of the FOV associated with the second image stream 5020G is , Circular boundaries can be formed.

  FIG. 26H schematically illustrates an exemplary arrangement of images that may be presented to a user in accordance with some other embodiments. As in Figures 26E-26G, the grid squares in Figure 26H graphically represent image points defined in two-dimensional angular space. Here, both the outer perimeter of the FOV associated with the first image stream 5010H and the outer perimeter of the FOV associated with the second image stream 5020H can form a circular boundary. In some other embodiments, one or both of the perimeter of the FOV associated with the first image stream 5010H and the perimeter of the FOV associated with the second image stream 5020H is an elliptical boundary or other. A shape can be formed. In some embodiments, the image source of the AR system of FIG. 26H is scanned in a predetermined pattern to produce a light beam for the first image stream 5010H and the second image stream 5020H with the desired boundary shape. May be included, which may include a scanning fiber.

  FIG. 27 illustrates a field of view 3002 and an eye view 3004 as shown in FIG. 24 overlaid on one of the displays 4052 in a wearable display device 4050 as shown in FIG. 25A. According to some embodiments, the wide FOV and low resolution first image stream 5010E illustrated in FIGS. 26E-26F can be displayed across the entire area of the display 4052 (first image). The relatively low resolution of stream 5010E is shown with a sparse and dense grid), while the narrow FOV and high resolution second image stream 5020E may be displayed in the user's current foveated region 3006. Yes (the relatively high resolution of the second image stream 5020E is shown with a dense grid). In FIG. 27, the first image stream 5010E and the second image stream 5020E are shown as being displayed in the “plane” of the display 4052, but in a see-through augmented reality (AR) display system, the first image stream The stream 5010E and the second image stream 5020E can also be presented to the user as a light field within an angular field of view. Such AR display systems can produce a display plane that appears to be "floating" at some distance in front of the user (eg, 2 meters). The display plane can appear much larger than the eyeglasses. This floating distance display is used to overlay information about the real world.

  28A-28B illustrate some of the principles described in FIGS. 26A-26D using example virtual content that may be presented to a user, according to some embodiments. Thus, FIGS. 28A-28B may represent a viewer and an AR system at the first and second time series stages, respectively. Further, some or all of the components shown in FIGS. 28A-28B may be the same or at least similar to the components as described above with reference to FIGS. 26A-26D.

  The AR system of FIGS. 28A-28B creates or dynamically repositions and / or reorients a head tracking virtual camera similar to the head tracking virtual cameras described above with reference to FIGS. 26A-26D, Virtual content may be rendered from the perspective of a head-tracking virtual camera, and light representative of the rendering of the virtual content may be projected through the eyepiece 6000 onto the retina of the viewer's eye 210. The AR system of FIGS. 28A-28B also creates or dynamically repositions and / or reorients a foveal tracking virtual camera similar to that described above with reference to FIGS. 26A-26D. , Virtual content may be rendered from the perspective of the foveal tracking virtual camera, and light representative of the rendering of the virtual content may be projected through the eyepiece 400 onto the fovea of the viewer's eye 210. As shown in FIGS. 28A-28B, such virtual content may include 3-D virtual objects 6011, 6012, and 6013. In some implementations, the AR system of FIGS. 28A-28B may be described with respect to one or more of the operations described immediately above with respect to the head-tracking rendering viewpoint, and the foveal tracking rendering viewpoint. One or more of these actions may be performed simultaneously. In other embodiments, the AR system of FIGS. 28A-28B may perform such operations at high speed and continuously.

  In this example, the FOV of the head-tracking rendering viewpoint employed by the AR system in FIGS. 28A-28B is to diagonally, horizontally, and / or vertically each of virtual objects 6011, 6012, and 6013. It can be an angular space wide enough to contain. For purposes of example, the position and orientation of the viewer's head are depicted in FIGS. 28A and 28B, respectively, such that the position and orientation of the head tracking rendering viewpoint remains the same throughout the two stages, respectively. May be treated as static throughout the first and second stages as described. The FOV of the head-tracking rendering viewpoint employed by the AR system is wide enough to contain the virtual objects 6011-6013 so that it is at least diagonally, horizontally, and / or vertically in α + ζ angle units. Must reach the range. More specifically, it can be seen that in the example of FIGS. 28A-28B, virtual objects 6011, 6012, and 6013 can span α-β, γ + δ, and ζ-ε angle unit ranges, respectively.

  In FIG. 28A, the viewer's eye 210 is oriented in a first manner with respect to the eyepiece 6000 so that the viewer may be able to see the eyepiece 6000 in a relatively straight-ahead direction. The orientation of the viewer's eye 210 in FIG. 28A may be the same or similar to the orientation of the viewer's eye 210, eg, as described above with reference to FIGS. It may be determined by the AR system using one or more of the techniques described in the book. Thus, at the stage depicted in FIG. 28A, the AR system is configured for head tracking and foveal tracking rendering viewpoints at relative positions and orientations similar to those of head tracking and foveal tracking rendering viewpoints 5010 and 5020A, respectively. May be adopted. In the particular example of FIG. 28A, the foveal tracking rendering viewpoint FOV employed by the AR system may include, for example, virtual object 6012, but not both virtual objects 6011 and 6013. In FIG. 28A, the AR system may have rendered a high resolution virtual object 6012 as would be captured from the foveal tracking virtual camera perspective and would have been captured from the head tracking virtual camera perspective. Virtual objects 6011 and 6013 can be rendered at a lower resolution. In addition, the AR system may project light representing such a rendering of virtual objects 6011, 6012, and 6013 through eyepiece 6000 onto the retina of viewer's eye 210. In some embodiments, the AR system may also render a lower resolution virtual object 6012 as it would be captured from the perspective of the head-tracking virtual camera.

  FIG. 28A also illustrates an exemplary light field 6030A, which is externally coupled by the eyepiece 6000 and projected onto the retina of the viewer's eye 210. Light field 6030A may include various angular light components that represent one or more of the renderings of virtual objects 6011, 6012, and 6013 described above. For example, the angular light component of the light field 6030A, which represents a virtual object 6011 as it would be captured from the perspective of a head-tracking virtual camera, spans −α to −β angular units with respect to the viewer's eye 210. The angular light of the light field 6030A, which may include what would be projected onto the retina of the viewer's eye 210 at an angle, and represents a virtual object 6013 as would be captured from the perspective of the head tracking virtual camera. The components may include those that will be projected onto the retina of the viewer's eye 210 at angles that range from ε to ζ angular units relative to the viewer's eye 210. Similarly, the angular light component of the light field 6030A, representing the virtual object 6012 as it would be captured from the perspective of the foveal tracking virtual camera, spans −γ to δ angular units with respect to the viewer's eye 210. It may include what would be projected onto the fovea of the viewer's eye 210 at an angle. Therefore, the component of the light field 6030A representing the virtual object 6012 (that is, the component that will be projected at an angle ranging from −γ to δ angular units with respect to the viewer's eye 210) represents the virtual object 6011 or 6013. , Light field 6030A components (i.e., components that will be projected to the viewer's eye 210 at angles ranging from-[alpha] to-[beta] or [epsilon] to [zeta] angle units) more densely dispersed in angular space. .. As such, the resolution with which the virtual object 6012 can be rendered and presented to the viewer can be higher than the resolution with which the virtual object 6011 or 6013 can be rendered and presented to the viewer.

  28B, the viewer's eye 210 is oriented relative to the eyepiece 6000 in a second manner, which is different from the first manner in which the viewer's eye 210 is oriented relative to the eyepiece 6000 in FIG. 28A. . The orientation of the viewer's eye 210 in FIG. 28B may be the same or similar to the orientation of the viewer's eye 210, eg, as described above with reference to FIGS. 26C-26D, and may include sensing components and / or the present specification. It may be determined by the AR system using one or more of the techniques described in the text. Thus, at the stage depicted in FIG. 28B, the AR system is performing a head-tracking and foveal-tracking rendering viewpoint at relative positions and orientations similar to those of the head-tracking and foveal-tracking rendering viewpoints 5010 and 5020C, respectively. May be adopted. In the particular example of FIG. 28B, the foveal tracking rendering viewpoint FOV employed by the AR system may include, for example, virtual object 6013, but not both virtual objects 6011 and 6012. In FIG. 28B, the AR system may render a virtual object 6013 at high resolution as would be captured from the foveal tracking virtual camera perspective, and would be captured from the head tracking virtual camera perspective. Virtual objects 6011 and 6012 can be rendered at a lower resolution. In addition, the AR system may project light representing such a rendering of virtual objects 6011, 6012, and 6013 through eyepiece 6000 onto the retina of viewer's eye 210. In some embodiments, the AR system may also render a lower resolution virtual object 6013 as would be captured from the perspective of the head tracking virtual camera.

  28B also illustrates an exemplary light field 6030B, which is externally coupled by the eyepiece 6000 and projected onto the retina of the viewer's eye 210. Light field 6030B may include various angular light components that represent one or more of the renderings of virtual objects 6011, 6012, and 6013 described above. For example, the angular light component of the light field 6030B, representing the virtual object 6011 as it would be captured from the perspective of the head-tracking virtual camera, spans -α to -β angular units with respect to the viewer's eye 210. Angular light of the light field 6030B, which may include what would be projected onto the retina of the viewer's eye 210 at an angle, and represents a virtual object 6012 as would be captured from the perspective of the head tracking virtual camera. The components may include those that will be projected onto the retina of the viewer's eye 210 at angles ranging from −γ to δ angular units with respect to the viewer's eye 210. Similarly, the angular light component of the light field 6030B, which represents the virtual object 6013 as it would be captured from the perspective of the foveal tracking virtual camera, has an angle ranging from ε to ζ angular units relative to the viewer's eye 210. , Which would be projected onto the fovea of the viewer's eye 210. Therefore, the component of the light field 6030B that represents the virtual object 6013 (that is, the component that will be projected at an angle ranging from ε to ζ angle units to the viewer's eye 210) represents the virtual object 6011 or 6012. , The component of the light field 6030A (ie, the component that is projected at an angle ranging from −α to −β or −γ to δ angular units with respect to the viewer's eye 210) is more densely dispersed in angular space. obtain. As such, the resolution with which the virtual object 6013 can be rendered and presented to the viewer can be higher than the resolution with which the virtual object 6011 or 6012 can be rendered and presented to the viewer. In fact, in the steps of FIGS. 28A to 28B, the AR system described herein with reference to it changes the visual line of sight of the eye 402 of the viewer between steps, from which the virtual content has high resolution. Effectively reorienting the viewable point.

  28C-28F illustrate some of the principles described in FIGS. 3E-3F using some example images that may be presented to a user, according to some embodiments. In some embodiments, one or more of the images and / or image streams depicted in FIGS. 28C-28F may include one or more of the depth planes described above with reference to FIG. 25B. May represent a two-dimensional image or part thereof to be displayed in a particular depth plane. That is, such images and / or image streams may represent 3-D virtual content projected onto at least one two-dimensional surface at a fixed distance from the user. In such an embodiment, such images and / or image streams may include one or more light fields with some angular field of view similar to those described above with reference to FIGS. 26A-26D and 28A-28B. It should be understood that may be presented to the user as

  As depicted, the first image stream 6010 includes trees. During the first time period represented by FIG. 28C, the eye tracking sensor causes the eye gaze of the user (ie, foveal vision) to fit within the first region 6010-1 of the tree, which includes the trunk of the tree. You can determine that you are being irritated. Responsive to determining that the user's line of sight is in focus in the first region 6010-1, the high resolution image associated with the first region 6010-1 of the first image stream 6010 is displayed. A second image stream 6020, including, can be positioned within the first region 410-1 at the same time as the display of the first image stream 6010. The first image stream 410 may have a lower resolution than the second image stream 6020, as illustrated in Figure 28C.

  During the second period of time, represented by FIG. 28D, the eye tracking sensor causes the eye gaze of the user to include a second region 6010-2 of the tree, where the user's line of sight includes tree branches, as illustrated in FIG. 28D. You can decide that you have moved to. Therefore, the second image stream 420 may be shifted to the second region 6010-2, changing its content to correspond to the content in the second region 6010-2 of the first image stream 6010. it can. The higher resolution second image stream 6020 overlays a portion of the first image stream 6010 within the fovea view of the user, thus lowering the first image stream 6010 surrounding the second image stream 6020. The portion of resolution cannot be perceived or perceived by the user. In this way, the user may perceive a combination of the first image stream 6010 and the second image stream 6020 as having both wide FOV and high resolution. Such a display system can offer several advantages. For example, a display system may maintain a relatively small form factor and keep computational resource requirements relatively low while providing a better user experience. The small form factor and low computational resource requirements may be due to the device only needing to generate high resolution images within a limited area of the display.

  The second image stream 6020 can be overlaid on the first image stream 6010, either simultaneously or in rapid succession. As discussed above, in some embodiments, the subset of content of the first image stream 6010 overlaid by the second image stream 6020 may have a more uniform brightness and a better resolution perception. It can be turned off or presented at a lower intensity. Also, in some embodiments, the second image stream associated with the second image stream 6020 may otherwise differ from the first image stream associated with the first image stream 6010. Please note. For example, the color resolution of the second image stream may be higher than the color resolution of the first image stream. The refresh rate of the second image stream may also be higher than the refresh rate of the first image stream.

  28E illustrates an exemplary high FOV low resolution image frame (ie, first image stream), and FIG. 28F illustrates an exemplary low FOV high resolution image frame (ie, second image stream), according to some embodiments. Image stream). As illustrated in FIG. 28E, region 6030 of the high FOV low resolution image frame that would be overlaid by the low FOV high resolution image frame may lack virtual content. By omitting the portion of the high FOV image corresponding to region 6030, any image blurring or blurring that results from slight differences in the two images can be avoided. The content of the low FOV high resolution image frame (eg, as illustrated in FIG. 28F) may include a high resolution version of the content corresponding to region 6030.

  FIG. 29A shows a simplified block diagram of a display system 7000A, according to some embodiments. The display system 7000A can include one or more sensors 7002 for detecting the position and movement of the user's head and the position of the user's eyes and the distance between the eyes. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and the like. In an augmented reality system, one or more sensors 7002 can be mounted on a head mounted frame.

  For example, in some implementations, the one or more sensors 7002 of the display system 7000A are part of a head-mounted transducer system and include one or more inertial transducers to move the user's head. An inertial measurement value indicative of may be captured. Thus, in these implementations, one or more sensors 7002 may be used to sense, measure, or collect information about the user's head movements. For example, such may be used to detect measured movements, velocities, accelerations, and / or positions of the user's head.

  In some embodiments, one or more sensors 7002 can include one or more front-facing cameras that can be used to capture information about the environment in which the user is located. A front facing camera may be used to capture information indicative of the user's distance and orientation to the environment and specific objects within the environment. When worn on the head, the front-facing camera is particularly suitable for capturing information indicative of the distance and orientation of the user's head with respect to the environment in which the user is located and specific objects within that environment. . A front-facing camera can be employed to detect head movement, speed of head movement, and acceleration. The front-facing camera can also be employed to detect or infer a center of user attention, for example, based at least in part on the orientation of the user's head. The orientation may be detected in any direction (eg, up, down, left and right with respect to the user's reference frame).

  One or more sensors 7002 may also include a pair of rear-facing cameras to track user eye movements, blinks, and depth of focus. Such eye tracking information can be determined, for example, by projecting light on the user's eye and detecting the return or reflection of at least a portion of the projected light. Further details on discussing eye tracking devices are provided in US Provisional Patent Application No. 61 / 801,219 entitled "DISPLAY SYSTEM AND METHOD", entitled "METHODS AND SYSTEM FOR CREATING FOCAL PLANES IN VIRTUAL AND AUGMENTED". US Provisional Patent Application No. 62 / 005,834, US Provisional Patent Application No. 61 / 776,771 entitled "SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY", and "METHOD AND SYSTEM FOR EYE TRACKING RINGS USING". US Provisional Patent Application No. 62 / 420,292, which is expressly incorporated herein by reference.

  Display system 7000A can further include a user orientation determination module 7004, which is communicatively coupled to one or more sensors 7002. User orientation determination module 7004 receives data from one or more sensors 7002 and uses such data to determine a user's head pose, corneal position, interpupillary distance, and the like. The user orientation determination module 7004 may detect the instantaneous position of the user's head and predict the position of the user's head based on position data received from one or more sensors 7002. The user orientation determination module 7004 also tracks the user's eyes based on tracking data received from one or more sensors 7002.

  Display system 7000A may further include a control subsystem, which may take any of a wide variety of forms. The control subsystem may be a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), application specific integrated circuits (ASICs), etc. It includes other integrated circuit controllers, programmable gate arrays (PGAS), eg, field PGAS (FPGAS), and / or programmable logic controller (PLU).

  In the example depicted in FIG. 29A, display system 7000A includes a central processing unit (CPU) 7010, a graphics processing unit (GPU) 7020, and frame buffers 7042 and 7044. In essence, as described in more detail below, CPU 7010 controls overall operation, while GPU 7020 renders frames from 3D data stored in database 7030 (ie, 3D scene These frames are stored in frame buffers 7042 and 7044. Although not shown, one or more additional integrated circuits read and / or read frames from and into frame buffers 7042 and 7044, image multiplexing subsystem 7060, foveal tracking beam steering component 7080. , And the like, and the operation of one or more other components of display system 7000A, such as equivalent components. Reading frames into and / or from frame buffers 542 and 544 may employ dynamic addressing, eg, the frames are overrendered. Display system 7000A further comprises read only memory (ROM) and random access memory (RAM). Display system 7000A further comprises a three-dimensional database 7030 from which GPU 7020 can access three-dimensional data of one or more scenes for rendering a frame.

  The CPU 7010 may include a high FOV low resolution rendering viewpoint determination module 7012 and a low FOV high resolution rendering viewpoint determination module 7014. In some embodiments, user orientation determination module 7004 can be part of CPU 7010.

  The high FOV low resolution rendering viewpoint determination module 7012 is a logic for mapping the data output by the user orientation determination module to locations and angles in 3D space from which the high FOV low resolution image will be perceived. Can be included. That is, the CPU 7010 determines the viewpoint of the virtual camera fixed with respect to the user's head at any given time based on the data received from the user orientation determination module 7004. In the context of the embodiments described above with reference to FIGS. 26A-26D and 28A-28B, the high FOV low resolution rendering viewpoint determination module 7012 is configured to determine the head position and orientation as shown by the user orientation determination module 7004. May be responsible for controlling the position and orientation of at least the head tracking virtual camera in the rendering space.

  The low FOV high resolution rendering viewpoint determination module 7014 will output the data output by the user orientation determination module (eg, data indicating the user's gaze and foveal location) from which a low FOV high resolution image will be perceived. Logic may be included to map to locations and angles in 3D space. That is, the CPU 7010 determines the viewpoint of the virtual camera that is fixed with respect to the fovea of the user at any given time based on the data received from the user orientation determination module 7004. In the context of the embodiments described above with reference to FIGS. 26A-26D and 28A-28B, the low FOV high resolution rendering viewpoint determination module 7014 monitors eye gaze as illustrated by the user orientation determination module 7004. Optionally, it may serve to control the position and orientation of at least the foveal tracking virtual camera in the rendering space.

  Display system 7000A may further include a graphics processing unit (GPU) 7020 and a database 7030. The database 7030 can store 3D virtual content. The GPU 7020 can access the 3D virtual content stored in the database 7030 to render the frame. The GPU 7020 determines the frame of virtual content to be low FOV and high from a virtual camera perspective (eg, foveal tracking rendering perspective) that is fixed relative to the user's fovea as determined by the CPU 7010 and provided as output. It can be rendered with resolution. The GPU 7020 also determines the frame of virtual content from the perspective of the virtual camera (eg, head tracking / non-fovealized perspective) that is fixed relative to the user's head as determined by the CPU 7010 and provided as output. Can be rendered with high FOV and low resolution. Further details discussing the creation, adjustment, and use of virtual cameras in the rendering process can be found in US patent application Ser. Are hereby incorporated by reference in their entirety).

  Frames rendered at high FOV low resolution of virtual content may be stored in a buffer 7042 of frames rendered at high FOV low resolution. Similarly, low FOV high resolution rendered frames of virtual content can be stored in a buffer 7044 of low FOV high resolution rendered frames. In some embodiments, high FOV low resolution rendered frame buffer 7042 and low FOV high resolution rendered frame buffer 7044 may be part of GPU 7020.

  Display system 7000A may further include an image multiplexing subsystem 7060 and an image multiplexing subsystem controller 7050 communicatively coupled to image multiplexing subsystem 7060. Image multiplexing subsystem 7060 multiplexes image source 7062 with a high FOV low resolution image frame and a low FOV high resolution image frame, as substantially described in further detail below with reference to FIGS. 30A-30B. And a multiplexing component 7064 for multiplexing. The image source 7062 can include, for example, a light source in combination with fiber scanning components, liquid crystal on silicon (LCoS), MEMS scanning mirrors, and the like. Multiplexing component 7064 can include optical elements such as polarization rotators, switchable optics, liquid crystal arrays, variable focus lenses, and the like. The multiplexing component 7064 can be internal or external to the image source 7062.

  Image multiplexing subsystem controller 7050 is communicatively coupled to image multiplexing subsystem 7060, high FOV low resolution rendered frame buffer 7042, and low FOV high resolution rendered frame buffer 7044. The control circuitry may send control signals to the image source 562 so that the appropriate image content is presented from each rendering perspective, as discussed above. The image multiplexing subsystem controller 7050 can also control the multiplexing component 7064 in conjunction with the image source 7062 to provide a multiplexed image stream.

  Display system 7000A may further include a foveal tracking beam steering component 7080 and a foveal tracking controller 7070 communicatively and / or operably coupled to foveal tracking beam steering component 7080. Foveal tracking controller 7070 receives output data (eg, as determined by low FOV high resolution rendering perspective determination module 7014 and / or user orientation determination module 7004) from CPU 7010 regarding the location of the fovea of the user, Such data can be used to control the position of the foveal tracking beam steering component 7080. The foveal tracking beam steering component 7080 dynamically steers the low FOV high resolution portion of the multiplexed image stream (produced by the image source 7062 and the multiplexing component 7064) towards the user's fovea. It can play a different role. Such a low FOV high resolution portion of the image stream may represent, for example, virtual content as would be captured from the perspective of the foveal tracking virtual camera.

  Display system 7000A also includes computer-readable instructions, a database, and a storage medium for storing CPU 7010, GPU 7020, and / or one or more other modules or other information usable by the controller of display system 7000A. You can Display system 7000A may further include input-output (I / O) interfaces, such as buttons, that a user may use for interaction with the display system. Display system 7000A may also include a wireless antenna for wireless communication with another portion of display system 7000A or the Internet.

  FIG. 29B schematically illustrates a cross-sectional view of an AR system 7000B, according to some embodiments. AR system 7000B, according to some embodiments, can incorporate at least some of the components of display system 7000A, as described above with reference to FIG. 29A, as shown in FIG. 25A. In addition, the wearable display device 4050 can be fitted into one of the displays 4052. For example, AR system 7000B can include an image multiplexing subsystem 560, which can include image source 7062 and one or more multiplexing components. In addition, the AR system 7000B can also include a foveal tracking beam steering component 7080, which in this example may be an electromechanical optical device such as a MEMS scanning mirror. Like the display system 7000A, the image multiplexing subsystem 7060 may be communicatively and / or operably coupled to an image multiplexing subsystem controller, wherein the foveal tracking beam steering component 7080 includes a foveal tracking controller. May be communicatively and / or operably coupled to. The AR system 7000B can further include one or more incoupling gratings (ICG) 7007 and one or more eyepieces 7008. Each incoupling grating 7007 can be configured to couple the first light beam and the second light beam into a separate eyepiece 7008. Each eyepiece 7008 can include an outcoupling grating for outcoupling the first light beam and the second light beam into the user's eye. The incoupling grating 7007 and the eyepiece 7008 may be referred to herein as the "visual assembly." It should be appreciated that various incoupling gratings (ICGs) disclosed herein may correspond to the incoupling optical elements 700, 710, 720 of FIGS. 9A-9C.

  30A-30B schematically illustrate a display system 8000 for projecting an image onto a user's eye, according to some embodiments. Display system 8000 includes an image source 8010. Image source 8010 projects a first light beam 8052 associated with a first image stream as shown in FIG. 30A and a second light beam associated with a second image stream as shown in FIG. 30B. It can be configured to project two light beams 8054. It is noted that the first light beam 8052 and the second light beam 8054 are depicted in FIGS. 30A-30B as schematic rays, which are not intended to represent rays that trace exact rays. I want to. The first light beam 8052 can be angularly expanded to cover a wider FOV, resulting in a lower angle resolution image stream. The second light beam 8054 can have a narrower FOV with higher angular resolution, as discussed above with reference to FIGS. 26A-26F and 28A-28D.

  The image source 8010 may include a liquid crystal on silicon (LCoS or LCOS) display (also referred to as a spatial light modulator), a scanning fiber, or a scanning mirror, according to various embodiments. For example, the image source 8010 may include a scanning device that scans the optical fiber in a predetermined pattern in response to the control signal. The predetermined pattern may correspond to some desired image shape, such as a rectangular or circular shape.

  According to some embodiments, the first light beam 8052 associated with the first image stream and the second light beam 8054 associated with the second image stream are multiplexed by the image source 8010. It can be output as a combined light beam. For example, polarization division multiplexing, time division multiplexing, wavelength division multiplexing, and the like, for multiplexing a light beam associated with a first image stream and a light beam associated with a second image stream. Can be used for.

  In an embodiment where polarization division multiplexing is used, the first light beam 8052 can be in a first polarization state and the second light beam 8054 can be in a second polarization state different from the first polarization state. It can be in a polarized state. For example, the first polarization state can be linear polarization oriented in a first direction and the second polarization state can be linear polarization oriented in a second direction orthogonal to the first direction. Can be In some other embodiments, the first polarization state can be left-hand circular polarization and the second polarization state can be right-hand circular polarization, or vice versa. it can. The first light beam 8052 and the second light beam 8054 can be projected by the image source 8010 simultaneously or sequentially.

  The display system 8000 can further include a polarizing beamsplitter (PBS) 8030 configured to demultiplex the first light beam 8052 from the second light beam 8054, according to some embodiments. . The polarizing beam splitter 8030 reflects a first light beam 8052 along a first optical path toward a viewing assembly, as shown in FIG. 30A, and a second beam of light, as shown in FIG. 30B. The light beam 8054 can be configured to be transmitted along the second optical path.

  An alternative to polarizing beam splitter 8030 may also be used to demultiplex the light beam. By way of example, beam splitters described herein, including but not limited to the polarizing beam splitter 8030 of FIGS. 30A and 30B, are replaced with switchable reflectors, such as liquid crystal switchable reflectors, or It may be implemented with it. In embodiments with such a switchable reflector, all other aspects disclosed herein may apply, except that the polarizing beam splitter is replaced by a switchable reflector. As an example, a switchable reflector such as switchable reflector 50042 of FIG. 53A can switch between a reflective state and a transmissive state in response to a control signal. By coordinating the switching of the switchable reflectors, the switchable reflectors can operate to demultiplex the light beam. As an example, the switchable reflector may be made reflective at any time when the first light beam is incident on the switchable reflector, and when the second light beam is incident on the switchable reflector, From time to time it may be made transparent, thus allowing demultiplexing of the first and second light beams. In some embodiments, the switchable reflector may be positioned at an angle (eg, a 45 ° angle) with respect to the light beams 8052, 8054. As a result, in the transmissive state one of the light beams 8052, 8054 is transmitted through the switchable reflector and in the reflective state the other of the light beams 8054, 8052 is the same as the light beam transmitted through the reflector. It is reflected as it travels away from the switchable reflector in different directions.

  Referring to FIG. 30B, the display system 8000 can further include a scanning mirror 8060 positioned downstream of the polarizing beam splitter 8030 along the second optical path. Scanning mirror 8060 is configured to reflect a second light beam 8054, which is to be projected to the user's eye, toward a viewing assembly. According to some embodiments, the scanning mirror 8060 may be controlled based on the fixation position of the user's eye to dynamically project the second image stream. For example, the scanning mirror 8060 can be in electrical communication with an eye gaze tracker that tracks user eye movements via a control circuit. The control circuitry sends a control signal to cause the second light beam 8054 to project the second image stream onto an area determined to cover the fovea of the user, and based on the user's current fixation point. Scanning mirror 8060 can be tilted and / or translated. In some embodiments, the scanning mirror 8060 can be a microelectromechanical system (MEMS) scanner with two degrees of freedom (ie, it can be scanned in two independent angles).

  In some other embodiments, instead of using scan mirror 8060, display system 8000 can use fixed mirrors. The step of controlling the position of the second image stream can be accomplished by laterally displacing the third optical lens 8046 (see description of the third optical lens 8046 below). For example, the third optical lens 8046 can be displaced up and down and towards and away from the page, as indicated by the arrow, to shift the position of the second image stream in two dimensions.

  In some embodiments, the display system 8000 can further include a polarization rotator 8022 positioned between the polarizing beam splitter 8030 and the scanning mirror 8060. The polarization rotator 8022 rotates the polarization of the second light beam 8054 so that the second light beam may have substantially the same polarization as that of the first light beam 8052 as it enters the viewing assembly. Can be configured as. The polarization rotator 8022 can include, for example, a half wave plate.

  In some embodiments, the display system 8000 can further include a first relay lens assembly for the first optical path and a second relay lens assembly for the second optical path. A first relay lens assembly is disposed between the image source 8010 and the polarizing beam splitter 8030, and a first optical lens 8042 is disposed downstream of the polarizing beam splitter 8030 along a first optical path. , A second optical lens 8044. The second relay lens assembly can include a first optical lens 8042 and a third optical lens 8046 disposed downstream of the polarizing beam splitter 8030 along the second optical path.

  FIG. 30C schematically illustrates a cross-sectional view of an augmented reality (AR) system, according to some embodiments. The AR system, according to some embodiments, can be fitted into one of the displays 4052 within the wearable display device 4050, as shown in FIG. 25A. The AR system can include a light projector 8000 for projecting a first light beam associated with the first image stream and a second light beam associated with the second image stream. Light projector 8000 can be similar to the display system illustrated in FIGS. 30A-30B. The AR system can further include one or more internal coupling grating (ICG) 8070 and one or more eyepieces 8080. Each incoupling grating 8070 can be configured to couple the first light beam and the second light beam into a separate eyepiece 8080. Each eyepiece 8080 can include an outcoupling grating for outcoupling the first light beam and the second light beam into the user's eye. The incoupling grating 8070 and the eyepiece 8080 may be referred to herein as the "visual assembly."

  FIG. 30D shows a simplified block diagram of a display system, according to some embodiments. The display system can include an image source 8010 and a scanning mirror 8060, substantially as described above with reference to FIGS. 30A-30C. The display system can also include an eye gaze tracker 8071 and a control circuit 8081. The control circuit 8081 can be communicatively coupled to the image source 8010, the scanning mirror 8060, and the eye gaze tracker 8071. The control circuit 8081 controls the current state of the user as determined by the eye gaze tracker 8071 so that the second light beam 8054 projects the second image stream into the area determined to cover the fovea of the user. A control signal may be sent to tilt and / or translate the scanning mirror 8060 based on the fixation point. The control circuit 8081 can also send control signals to the image source 8010 so that the appropriate image content is presented to the first image stream and the second image stream, as discussed above. .. The display system also stores a central processing unit (CPU) 8096, a graphics processing unit (GPU) 8098, computer readable instructions, a database, and control circuitry 8081, a CPU 8096, and other information usable by the GPU 8098. A storage medium 8090 can be included. The display system can further include an input-output (I / O) interface 8092, such as a button, that a user can use to interact with the display system. The display system may also include a wireless antenna 8094 for wireless communication with another part of the display system or the Internet. The display system may also include other sensors such as cameras.

FIG. 31A schematically illustrates the operating principle of a first relay lens assembly, according to some embodiments. The first relay lens assembly can operate in a manner similar to a telescope. A collimated first light beam 8052 associated with the first image stream is incident on the first optical lens 8042 at an angle of incidence θ _A , and the first optical lens 8042 causes the first optical lens 8042 to approximate the first optical lens. It is focused on the real image point P ₀ located at the focal plane of 8042. The real image point P ₀ is also located approximately at the focal plane of the second optical lens 8044. Therefore, the first light beam 8052 emitted from the real image point P ₀ is collimated by the second optical lens 80044 and exits from the second optical lens 8044 at a transmission angle of θ _B.

The ratio of θ _B and θ _A may cause a _first angular magnification factor M ₁ .
Is. The magnitude of the first angular magnification M ₁ may be approximately equal to the ratio of the focal length of the first optical lens 8042f _{A and} the focal length of the second optical lens 8044f _B. Therefore,
Is. In some embodiments, the first relay lens assembly has a magnitude of the first angular magnification M ₁ of, for example,
Is configured to exceed 1. Thus, referring again to FIG. 30A, the collimated first light beam 8052 associated with the first image stream is angularly output by the first relay lens assembly as it exits the second optical lens 8044. It can be magnified, which is then projected onto the viewing assembly to present a first image stream with a relatively wide first field of view FOV ₁ .

FIG. 31B schematically illustrates the operating principle of a second relay lens assembly, according to some embodiments. The second relay lens assembly can also operate in a manner similar to a telescope. The collimated second light beam 8054 associated with the second image stream is incident on the first optical lens 8042 at an angle of incidence θ _A , and the first optical lens 8042 causes the first optical lens 8042 to approximate the first optical lens. It is focused on the real image point P ₀ located at the focal plane of 8042. The real image point P ₀ is also located approximately at the focal plane of the third optical lens 8046. Therefore, the second light beam 8054 emitted from the real image point P ₀ is collimated by the third optical lens 8046 and exits from the third optical lens 8046 at the transmission angle θ _C.

The ratio of θ _C and θ _A may give rise to a _second angular magnification factor M ₂ .
Is. The magnitude of the second angular magnification factor M ₂ may be approximately equal to the ratio of the focal length of the first optical lens 8042f _{A and} the focal length of the third optical lens 644f _C. Therefore,
Is. The second lens assembly can be configured such that the magnitude of the second angular magnification factor M ₂ is less than the first angular magnification factor M ₁ . In some embodiments, the second angular magnification factor M ₂ is, for example,
By having a value of 1 (ie no magnification) or less than 1 (ie demagnification). Thus, referring again to FIG. 30B, the collimated second light beam 8054 associated with the second image stream may have a _second field of view FOV ₂ as it exits the third optical lens 8046. Yes, the second field of view FOV ₂ is less than the first field of view FOV ₁ of the first light beam 8052 associated with the first image stream.

In FIG. 31A, the collimated first light beam 8052 has an initial beam width w _A when it is incident on the first optical lens 8042 and a final beam width when it exits the second optical lens 8044. has a w _B, the final beam width w _B it is noted that the narrower than the initial beam width w _a. Also, in FIG. 31B, the collimated second light beam 8054 has an initial beam width w _A when it is incident on the first optical lens 8042 and a final beam width when it exits the third optical lens 8046. has a beam width w _C, final beam width w _C it should be noted that it is approximately the same as the initial beam width w _a. In other words, the final beam width w _C of the second light beam 8054 is wider than the final beam width w _{B of} the first light beam 8052. The wider beamwidth will result in a sharper angular resolution perceived by the eye. This can be explained by Gaussian beam physics, where collimated beams with wider beam waists have lower angular divergence over propagation to infinity. Therefore, increasing the FOV may reduce the beamwidth and hence the angular resolution, which is consistent with the Lagrange invariant.

In some embodiments, the first angular magnification factor M ₁ can have a magnitude of about 3 and the second angular magnification factor M ₂ can have a magnitude of about 1. 30A-30B, a collimated first light beam 8052 associated with a first image stream and a collimated second light beam 8054 associated with a second image stream are image sources 8010. Suppose we have the same initial FOV of about 20 degrees projected by. The collimated first light beam 8052 exiting the second optical lens 644 may have a _first field of view FOV ₁ of about 60 degrees, while the collimated first light beam 8052 exits the third optical lens 8046. The second light beam 654 may have a second field of view FOV ₂ of about 20 degrees. In some embodiments, the first FOV can range from about 30 degrees to about 90 degrees and the second FOV can range from about 10 degrees to about 30 degrees.

  As illustrated in FIGS. 28C-28D, the second image stream 6020 can be a high resolution version of a portion of the first image stream 6010, on the wide FOV and low resolution first image stream 6010. Is overlaid on and properly matched to. The content of the second image stream 6020 is such that the content of the second image stream 6020 corresponds to the portion of the first image stream 6010 overlaid by the second image stream 6020. It changes as it shifts with respect to the first image stream 6010. Since the second image stream 6020 continuously covers the fovea of the user, the user can view the first image stream 6010 and the second image stream as a composite image stream with both wide FOV and high resolution. 6020 combinations can be perceived.

  31C-31D schematically illustrate a display system 10000 according to some other embodiments. Display system 10000 includes an image source 9010 and a beam splitter 9030. The image source 9010 can provide a first light beam 8052 associated with the first image stream and a second light beam 8054 associated with the second image stream. The first light beam 8052 and the second light beam 8054 can be time division multiplexed, polarization division multiplexed, wavelength division multiplexed, or the like. Beamsplitter 9030 acts as a demultiplexer, directing first light beam 8052 and second light beam 8054 into a first optical path and a second optical beam as depicted in FIGS. 31C and 31D, respectively. It can be separated towards the path.

  The display system 10000 can also include a first optical lens 9042 and a second optical lens 9044 disposed downstream of the beam splitter 9030 along the first optical path. The combination of the first optical lens 9042 and the second optical lens 9044 can serve as a first relay lens assembly for the first light beam 8052. In some embodiments, the first relay lens assembly can provide an angular magnification factor for the first light beam 8052 of greater than 1, as described above with respect to FIG. 31A.

  The display system 10000 can also include a third optical lens 9045 and a fourth optical lens 9046 disposed downstream of the beamsplitter 9030 along the second optical path. The combination of the third optical lens 9045 and the fourth optical lens 9046 can serve as a second relay lens assembly for the second light beam 8054. In some embodiments, the second relay lens assembly provides an angular magnification factor for the second light beam 8054 that is substantially 1 or less than 1 as described above with respect to FIG. 31B. can do.

  Display system 10000 can also include a scanning mirror 9060 positioned downstream of the second relay lens assembly along the second optical path. Scanning mirror 9060 is configured to reflect a second light beam 8054, which is to be projected to the user's eye, toward a viewing assembly. According to some embodiments, the scanning mirror 9060 can be controlled based on the fixed position of the user's eye to dynamically project the second image stream.

  The display system 10000 can also include a fifth optical lens 9047 and a sixth optical lens 9048 disposed downstream of the scan mirror 9060 along the second optical path. The combination of the fifth optical lens 9047 and the sixth optical lens 9048 can serve as a third relay lens assembly for the second light beam 8054. In some embodiments, the third relay lens assembly provides an angular magnification factor for the second light beam 8054 that is substantially 1 or less than 1 as described above with respect to FIG. 31B. can do.

  In some embodiments, the display system 10000 can also include a polarizer 9080 and a switching polarization rotator 9090. Image source 9010 can provide an unpolarized first light beam 8052 and an unpolarized second light beam 8054, which are time division multiplexed. First light beam 652 and second light beam 654 may be polarized after passing through polarizer 9080. The switching polarization rotator 9090 can be operated synchronously with the time division multiplexing of the first light beam 8052 and the second light beam 8054. For example, the switching polarization rotator 9090 is such that the polarization of the first light beam 8052 is unchanged after passing through the switching rotator 9090, while the polarization of the second light beam 8054 is passing through the switching polarization rotator 9090. , Rotated 90 degrees, or vice versa. Thus, the first light beam 8052 can be reflected by the polarizing beam splitter 9030 along the first optical path, as illustrated in FIG. 31C, and the second light beam 8054 can be polarized light. The splitter 9030 may allow transmission along the second optical path, as illustrated in FIG. 31D.

  32A-32C schematically illustrate a display system 10000 according to some other embodiments. In some examples, one or more components of display system 10000 may be the same or similar to one or more components of a display system as described above with reference to FIGS. 31C-31D. The display system 10000, in some embodiments, respectively includes elements 9010, 9030, 9042, 9044, 9045, 9046, 9047, 9048, 9060 of the display system as described above with reference to FIGS. 31C-31D, respectively. , 9080, and 9090, which may be the same as or similar to image source 10010, beam splitter 10030, first optical lens 10042, second optical lens 10044, third optical lens 10045, and fourth optical lens 10045. It includes a lens 10046, a fifth optical lens 10047, a sixth optical lens 10048, a scanning mirror 10060, a polarizer 10080, and a switching polarization rotator 10090.

  More specifically, FIGS. 32A-32C illustrate display system 10000 in each of three different stages. In each of the three stages, the image source 10010 is captured from the perspective of the foveal tracking virtual camera, and the range of angular light field components that represent virtual content as would be captured from the perspective of the head tracking virtual camera. And a range of angular light field components representing the virtual content as would be output. The two sets of angular light field components may be time division multiplexed, polarization division multiplexed, wavelength division multiplexed, or the like, for example. Therefore, the angular light field component associated with the head-tracking virtual camera is redirected upward by the polarizing beamsplitter 10030 along the first optical path through the first and second optical lenses 10042 and 10044. And the angular light field component associated with the foveal tracking virtual camera is passed through polarizing beam splitter 10030, along a second optical path, through third and fourth optical lenses 10045 and 10046 to scanning mirror 10060. Can be passed through and reflected upward through the fifth and sixth optical lenses 10047 and 10048.

  The virtual content represented by the angular light field component associated with the head tracking virtual camera may be rendered at a relatively low resolution upstream from the image source 10010, while the angular light field associated with the foveal tracking virtual camera. The virtual content represented by the components can be rendered at relatively high resolution upstream from the image source 10010. Also, as shown in FIGS. 32A-32C, the display system 10000 associates an angular light field component associated with a head-tracking rendering viewpoint and a foveal tracking rendering viewpoint as a high FOV and low FOV light field, respectively. And an output angular light field component. In each of FIGS. 32A-32C, the light field component propagating along the first optical path is output by the display system 10000 as a relatively wide cone of light 10052.

  At the stage depicted in FIG. 32A, scan mirror 10060 is in the first position. Thus, the light field component passing through the polarizing beam splitter 10030 and propagating along the second optical path is displayed by the display system 10000 as a relatively narrow cone of light 10054A that spans substantially the central region of the angular space. You can see that it is output. In the context of the embodiments described above with reference to FIGS. 28A-28B, display system 10000 may be configured such that, for example, a user's eye is oriented in a manner similar to that of viewer's eye 210 in FIG. 28A. , Scan mirror 10060 may be installed in the first position shown in FIG. 32A. As such, the light component 10054A may represent virtual content, such as virtual object 6012, in a relatively central region of the rendering space. 28A-28B, the relatively wide cone of light 10052 may include virtual content, such as virtual objects 6011 and 6013, in an off-center region of the rendering space. In some embodiments, the relatively broad cone of light 10052 may also include light components, such as represented by light component 10054A, but representing the same virtual content at a lower resolution.

  At the stage depicted in FIG. 32B, the scanning mirror 10060 is in a second position different from the first position. Thus, the light field component passing through the polarizing beamsplitter 10030 and propagating along the second optical path is compared by the display system 10000 with the light 10054B spanning one substantially off-centered region of angular space. It can be seen that the output is a narrow cone. In the context of the embodiments described above with reference to FIGS. 28A-28B, the display system 10000 may be such that the user's eye is the viewer's eye 210 while the viewer is looking at the virtual object 6011, for example. When oriented in a manner similar to, scan mirror 10060 may be placed in the second position shown in FIG. 32B. Thus, light component 10054B may represent virtual content, such as virtual object 6011, within one relatively off-center region of rendering space. In addition to the embodiments of FIGS. 28A-28B, the relatively wide cone of light 10052 allows virtual content, such as virtual object 6013, to appear within regions off-center of virtual content, such as virtual object 6013, in the rendering space. Content may be included within the central region of the rendering space. In some embodiments, the relatively wide cone of light 10052 further represents the same virtual content, such as represented by light component 10054B, but at a lower resolution. It may include a light component.

  At the stage depicted in FIG. 32C, the scanning mirror 10060 is in a third position different from the first and second positions. Thus, the light field component passing through the polarizing beam splitter 10030 and propagating along the second optical path of the light 10054C is spread by the display system 10000 to another different substantially decentered region of the angular space. It can be seen that the output is as a relatively narrow cone. In the context of the embodiments described above with reference to FIGS. 28A-28B, display system 10000 may be configured such that, for example, a user's eye is oriented in a manner similar to that of viewer's eye 210 in FIG. 28B. , Scan mirror 10060 may be installed in the second position shown in FIG. 32C. As such, light component 10054C may represent virtual content, such as virtual object 6013, in another relatively off-center region of the rendering space. In addition to the embodiments of FIGS. 28A-28B, the relatively wide cone of light 10052 causes the virtual content, such as virtual object 6011, to be offset from the center of the rendering space described above with reference to FIG. 32B. Virtual content such as virtual object 6012 may be included within the central region of the rendering space. In some embodiments, the relatively wide cone of light 10052 may also include light components, such as represented by light component 10054C, but representing the same virtual content at a lower resolution.

  33A-33B schematically illustrate a display system 11000 for presenting a first image stream and a second image stream, where time division multiplexing is performed on the first image stream, according to some embodiments. Used to multiplex a first light beam 8052 associated with a second light beam 8054 associated with a second image stream. Display system 11000 is similar to display system 8000. The image source 11010 can be configured to provide a time division multiplexed first light beam 8052 and a second light beam 8054. The first light beam 8052 and the second light beam 8054 can be in the same polarization state as that output from the image source 8010. It is noted that first light beam 8052 and second light beam 8054 are depicted in FIGS. 33A-33B as schematic rays, which are not intended to represent the exact ray traced rays. I want to.

  The display system 11000 can further include a switching polarization rotator 11020, the operation of which can be synchronized with time division multiplexing of the first light beam 8052 and the second light beam 8054. For example, the switching polarization rotator 11020 is such that the polarization of the first light beam 8052 is unchanged after passing through the switching rotator 11020, while the polarization of the second light beam 8054 is passing through the switching polarization rotator 11020. , Rotated 90 degrees, or vice versa. Accordingly, the first light beam 8052 can be reflected by the polarizing beam splitter 8030 along the first optical path, as illustrated in FIG. 33A, and the second light beam 8054 can be polarized light. The splitter 8030 may allow transmission along the second optical path, as illustrated in Figure 33B.

  In some other embodiments, the switching polarization rotator 11020 can be part of the image source 11010. In such a case, the first light beam 8052 and the second light beam 8054 are sequentially emitted, and the first light beam 8052 projected from the image source 8010 is polarized in the first direction and the image source 8010. The second light beam 8054 projected from will be polarized in the second direction.

  According to some embodiments, switching a first light beam 8052 associated with a first image stream and a second light beam 8054 associated with a second image stream when time division multiplexed. Possible mirrors can be used in place of the polarizing beam splitter 8030 shown in FIGS. 30A-30B, 31C-31D, and 33A-33B. Switching of the switchable mirror can be synchronized with time division multiplexing of the first light beam 8052 and the second light beam 8054. For example, the switchable mirror may have a first mirror such that the mirror operates to reflect a first light beam 8052 along a first optical path, as illustrated in FIGS. 30A, 31C, and 33A. Of the second light beam 8054 is switched to a first state for transmission of the second light beam 8054 along the second optical path as shown in FIGS. 30B, 31D, and 33B. The second state can be switched for the second light beam 8054 to operate as to.

  According to some embodiments, wavelength division multiplexing is used to multiplex a first light beam associated with a first image stream and a second light beam associated with a second image stream. Can be done. For example, the first light beam can consist of light within the first set of wavelength ranges for red, green, and blue light, and the second light beam for red, green, and blue light, It can consist of light within the second set of wavelength ranges. The two sets of wavelength ranges can be offset with respect to each other, but the combination of the second set of wavelength ranges is substantially the same as the white light produced by the combination of the first set of wavelength ranges. It produces the same, white light.

  When wavelength division multiplexing is used, the display system replaces the polarizing beamsplitter and includes a first light beam associated with the first image stream and a second light beam associated with the second image stream. A dichroic beamsplitter may be included to separate the beams. For example, a dichroic beam splitter has high reflectance and low transmittance values for a first set of wavelength ranges and low reflectance and high transmittance values for a second set of wavelength ranges. Can be configured as. In some embodiments, the first light beam and the second light beam can be projected in parallel without the need for a switchable polarization rotator.

  34A-34B schematically illustrate a display system 12000 according to some other embodiments. Display system 12000 includes an image source 12010. Image source 12010 includes a first light beam 12052 associated with a first image stream as illustrated in FIG. 34A and a second light beam 12052 associated with a second image stream as illustrated in FIG. 34B. The light beam 12054 can be configured to project. The first image stream can be a wide FOV and low resolution image stream and the second image stream can be a narrow FOV and high resolution image stream, as discussed above with reference to FIGS. 26E-26F. Can be The first light beam 12052 and the second light beam 12054 can be multiplexed using, for example, polarization division multiplexing, time division multiplexing, wavelength division multiplexing, and the like. In FIGS. 34A-34B, the first light beam 12052 and the second light beam 12054 are depicted as schematic rays, which are not intended to represent exact ray traced rays.

  The display system 12000 may further include a beam splitter 12030 configured to demultiplex the first light beam 12052 and the second light beam 12054, according to some embodiments. For example, the beam splitter 12030 can be a polarizing beam splitter (PBS) or a dichroic beam splitter. The beamsplitter 12030 reflects the first light beam 12052 along the first optical path as shown in FIG. 34A and the second light beam 12054 as the second light beam 12054 as shown in FIG. 34B. Can be configured to transmit along the optical path of the.

  Display system 12000 can further include switchable optical element 12040. Switchable optical element 12040 is shown as a single element, but can also include a pair of sub-switchable optical elements that function as a switchable relay lens assembly. Each sub-switchable optical element is switched to a first state or operates as an optical lens with a second optical power different from the first optical power so that it operates as an optical lens with a first optical power. The second state can be switched to, Therefore, the switchable optical element 12040 is configured such that when the sub-switchable optical element is switched to the first state, as shown in FIG. As illustrated at 34B, when switched to the first state, a second angular magnification ratio different from the first angular magnification ratio can be provided.

Each sub-switchable optical element can take many forms, including, for example, a liquid crystal variable focus lens, a tunable diffractive lens, or a deformable lens. In general, any lens that can be configured to change shape or configuration and adjust its refractive power may be applied. In some embodiments, each sub-switchable optical element has substantially a first optical power for light with a first polarization and a first optical power for light with a second polarization. Can be a multifocal birefringent lens having a different second power. For example, multifocal birefringent lens, such that the polymer exhibits a normal refractive index n _o and the surplus refractive index n _e, is the birefringence by the orientation process by which stretch the polymer under the conditions defined, A polymer can be included.

When the first light beam 12052 and the second light beam 12054 are time division multiplexed, the switching of the switchable optical element 12040 is performed by each sub-switchable optical element as shown in FIG. 34A. Acts as an optical lens with a first optical power for the light beam 12052 of FIG. 34, and acts as an optical lens with a second optical power for the second light beam 12054 as illustrated in FIG. 34B. Thus, the time division multiplexing of the first light beam 12052 and the second light beam 12054 can be synchronized. Thus, the first light beam 12052 associated with the first image stream can be angularly expanded by the switchable optical element 12040 as it exits the switchable optical element 12040, and subsequently relatively. Can be projected onto the viewing assembly to present a first image stream with a wide first field of view FOV ₁ .

  The display system 12000 can further include a first mirror 12060 positioned downstream from the beamsplitter 12030 along a second optical path, as illustrated in FIG. 34B. The first mirror 12060 can retroreflect the second light beam 12054 toward the beam splitter 12030, which in turn can be reflected by the beam splitter 12030 toward the second mirror 12070. it can.

The second mirror 12070 is positioned below the beamsplitter 12030 as illustrated in Figure 34B. The second mirror 12070 may retroreflect the second light beam 12054 toward the beam splitter 12030, which is subsequently transmitted by the beam splitter 12030 toward the switchable optical element 12040. You can As explained above, each sub-switchable optical element can be switched to a second state so that it can operate as an optical lens with a second optical power for the second light beam 12054. it can. The second optical power can be less than or substantially zero or negative in relation to the first optical power associated with the first state. Thus, the second light beam 12054 can be angularly expanded, unexpanded, or de-expanded by an amount less than the first light beam 12052 as it exits the switchable optical element 12040. Thus, the second light beam 12054 can subsequently be projected onto the viewing assembly to present a second image stream with a relatively narrow second field of view FOV ₂ .

  In some embodiments, the second mirror 12070 may be tilted in two directions, as illustrated in FIG. 34B, a two-dimensional (2D) scanning mirror (ie, two rotational degrees of freedom), such as a 2D MEMS scanner. Scanning mirror). Tilting the second mirror 12070 is controlled based on the fixation position of the user's eye so that the second light beam 12054 can project the second image stream into the fovea of the user. You can In some other embodiments, the second mirror 12070 can be a fixed mirror and the first mirror 12060 can be a 2D scanning mirror. In some further embodiments, the first mirror can be a one-dimensional (1D) scanning mirror (ie, a scanning mirror with one rotational degree of freedom) that can be tilted in a first direction, and a second Can be a 1D scanning mirror that can be tilted in a second direction.

  FIG. 35 schematically illustrates a display system 13000 according to some other embodiments. Display system 13000 includes an image source 13010. The image source 13010 outputs a first light beam associated with the first image stream in right-hand circular polarization (RHCP) and a second light beam associated with the second image stream in left-hand circular polarization (LHCP). ) (Or vice versa).

  Display system 13000 can further include a beam splitter 13030 configured to demultiplex the first light beam and the second light beam. For example, the beam splitter 13030 can comprise a liquid crystal material that reflects a right-handed circularly polarized first light beam and transmits a left-handed circularly polarized second light beam.

The display system 13000 can further include a first switchable optical element 13042 and a second switchable optical element 13044, the combination of which can serve as a relay lens assembly. The first switchable optical element 13042 and the second switchable optical element 13044 each have a first focal length f _RHCP for right-handed circularly polarized light and a second focal length f _LHCP for left-handed circularly polarized light. And a liquid crystal material can be provided. Therefore, the combination of the first switchable optical element 13042 and the second switchable optical element 13044 results in a first angular magnification factor for the first light beam and a second angular magnification factor different from the first angular magnification factor. A rate can be provided for the second light beam. For example, the first angular magnification can be greater than 1 and the second angular magnification can be equal to or less than 1.

  FIG. 36A schematically illustrates an augmented reality eyepiece display system 14000 according to some embodiments. FIG. 36A shows a portion of display system 14000 for one eye 210. In fact, a second such system would be provided for the other eye of the user. Two such systems, according to embodiments, are incorporated into augmented reality glasses. Referring to FIG. 36A, a red laser diode 14002 is optically coupled into a red light input surface 14006 of a red-green-blue (RGB) dichroic combiner cube 14008 through a red laser collimating lens 14004. The green laser diode 14010 is optically coupled into the green light input surface 14014 of the RGB dichroic combiner cube 14008 through the green laser collimating lens 14012. Similarly, the blue laser diode 14016 is optically coupled into the blue light input surface 14020 of the RGB dichroic combiner cube 14008 through the blue laser collimating lens 14018. The RGB dichroic combiner cube 14008 has an output surface 14022. The RGB dichroic combiner cube 14008 includes a red reflective dichroic mirror (short wavelength pass mirror) 14024 set at 45 degrees to reflect light from the red laser diode 14002 through the output surface 14022. The RGB dichroic combiner cube 14008 also has a blue reflective dichroic mirror (long wavelength pass mirror) set at 135 degrees (perpendicular to the red reflective dichroic mirror 14024) to reflect light from the blue laser diode 14016 to the output surface 14022. ) 14026. Light from the green laser diode 14010 passes (is transmitted) through the red reflective dichroic mirror 14024 and the blue reflective dichroic mirror 14026 to the output surface 14022. The red reflective dichroic mirror 14024 and the blue reflective dichroic mirror 14026 can be implemented as thin film optical interference films.

  Red, green, and blue laser diodes 14002, 14010, 14016 are separately modulated with red, blue, and green channel image information. Including a first cycle at which image information will be directed to the fovea of the user's retina and a subsequent cycle at which the image information will be directed to a larger portion of the user's retina. The cycle is repeated sequentially. There may be some degree of angular overlap between the image information directed to the user's retina in the first cycle and the image information directed to the user's retina during subsequent cycles of the cycle. In other words, some parts of the user's eyes may receive light during both cycles. Rather than attempting to achieve sharp boundaries, overlapping boundaries characterized by tapered strength may be used. An optical arrangement for achieving the above functionality will be described below.

  The dichroic combiner cube 14008 outputs a collimated beam 14028 that includes red, blue, and green components. The collimated beam 14028 is incident on the first two degree of freedom image scanning mirror 14030. The image scanning mirror 14030 has two degrees of freedom rotation and can be oriented at an angle within a predetermined angular range. Each orientation of image scanning mirror 14030 effectively corresponds to an angular coordinate in image space. The orientation of the image scanning mirror 14030 cooperates with the modulation of the red, green, and blue laser diodes 14002, 14010, 14016 based on the image information so as to present the image ultimately to the user's eye. To be scanned.

  The light deflected by the image scanning mirror 14030 is coupled to the polarization rotation switch 14034 through the first relay lens element 14032. Alternatively, the polarization rotation switch may be located closer to the laser diodes 14002, 14010, 14016. The polarization rotation switch 14034 is electrically controlled by an electronic device (not shown in FIG. 36A). The polarization rotation switch 14034 can be implemented as a liquid crystal polarization rotation switch. The polarization rotation switch 14034 receives light of a specific linear polarization output by the laser diodes 14002, 14010, 14016 and transmitted through the collimating lenses 14004, 14012, 14018 and the RGB dichroic combiner cube 14008 without modifying the polarization. To do. The polarization rotation switch 14034 either passes the incident light without modifying its polarization or rotates the polarization of the light by 90 degrees under the control of an external electrical signal.

  The light emitted from the polarization rotation switch 14034 is coupled to the polarization beam splitter (PBS) 14036. The PBS 14036 incorporates therein a polarization selective reflector 14038 diagonally arranged across the PBS 14036. The polarization selective reflector 14038 can be of a type that includes an array of parallel metal conductive lines (not visible in Figure 36A). Light polarized parallel to the metal conductive line (ie, having an electric field direction) is reflected and light polarized perpendicular to the conductive metal line is transmitted. For the embodiment shown in Figure 36A, the conductive metal lines are assumed to be oriented perpendicular to the plane of the drawing sheet. Using such an orientation, the polarization selective reflector 14038 will reflect S-polarized light and transmit P-polarized light.

  Considering the first case, in which the polarization rotation switch 14034 is in the state of outputting P-polarized light, such P-polarized light passes through the polarization-selective reflector 14038 and through the PBS 14036 to the first four. A quarter wave plate (QWP) 14040 will generally be reached. The first QWP 14040 is oriented to convert P-polarized light into right circularly polarized (RHCP) light. (Alternatively, the first QWP may be oriented to convert P-polarized light into LHCP, which, as will be apparent after reviewing the remaining description of FIG. Changes to the components will also be made.) After passing through the first QWP 14040, the light will reach the second relay lens element 14042. The first relay lens element 14032 and the second relay lens element 14042 are equal magnification afocal compound lenses. It should be noted that the image scanning mirror 14030 is spaced from the first relay lens element 14032 by a distance equal to the focal length of the first relay lens element 14032. The second relay lens element 14032 will re-collimate the light (first collimated by the collimating lenses 14004, 14012, 14018). Further, the light propagating from the second relay lens element 14042 intersects the optical axis OA near the point P1 separated from the second relay lens element 14042 by the focal length of the second relay lens element 14042. Please note that In the embodiment shown in FIG. 36A, the first relay lens element 14032 and the second relay lens element 14042 have the same focal length.

  After exiting the second relay lens element 14042, the light will be incident on the first group of positive refraction lenses 14044 of the first group 14046 of 2x afocal magnifying lens 14048. In addition to the first group of positive refractive lenses 14044, the first group 14046 also includes a first group of geometric phase lenses 14050. After passing through the first group of geometrical phase lenses 14050, the light includes a second group of positive refractive lenses 14054 and a second group of geometrical phase lenses 14056, a second group 14052. Pass through. The geometric phase lenses 14050, 14056 include pattern-matched liquid crystal material. Geometric phase lenses (also known as "polarization-oriented flat lenses") are available from Edmund Optics (Barrington, New Jersey). Geometric phase lenses 14050, 14056 are positive lenses for circularly polarized light in which they have a handedness (RH or LH) that matches their handedness, and for circularly polarized light of opposite handedness. It has the property of being a negative lens. Geometric phase lenses also have the property that when transmitting light, they reverse the handedness of circularly polarized light. In the embodiment shown in FIG. 36A, the geometric phase lenses 14050, 14056 are clockwise. Note that the system may be modified to accommodate use with counterclockwise geometric phase lenses.

In operation, when RHCP light is passed through the first group 14046, the first group of geometric phase lenses 14050 has a positive refractive power of the first group 14046 and a first group of refractive lenses 14044. As a single positive power less than the first group 14046 would have a focal length approximately equal to the distance from the principal plane of the first group 14046 to the point F _RHCP shown in FIG. 36A. , Will act as a negative lens. Propagating through the first group of geometric phase lenses 14050 will convert the light to the left circularly polarized (LHCP) state. For light in the LHCP state, the second group of geometric phase lenses 14056 has a positive refractive power, and thus the positive power of the second group 14052 has a positive refractive power of the second group. It would exceed the positive refractive power of 14054 alone. In this case, the focal length of the second group 14052 is also equal to the distance from the principal plane of the second group 14052 to the point F _RHCP , and the subscript “RHCP” refers to the polarization state of the light incident on the magnifying lens 14048. .. Since the point F _RHCP is closer to the second group 14052 than the first group 14046, the 2 × afocal magnifying lens 14048 defines a magnifying lens (1 for the RHCP light received from the second relay lens element 14042). Will have a higher expansion rate).

  Considering now the second case, in which the polarization rotation switch 14034 is in the state of outputting S-polarized light, such S-polarized light is reflected nominally 90 degrees by the polarization selective reflector 14038, and then , Through a second QWP 14058 and then through a third relay lens element 14060, which causes the light to be deflected towards a fixed mirror 14062. Note that for S-polarization, the first relay lens element 14032 is combined with the third relay lens element 14060 to form a unit magnification afocal relay system. The fixed mirror 14062 retroreflects the light through the third relay lens element 14060 and the second QWP 14058 and changes the sign, but does not change the absolute value of the angle of the light beam with respect to the optical axis OA. First, after passing through the second QWP 14058, the S-polarized light is converted into circularly polarized light of a particular handedness (either RHCP or LHCP by choosing the fast and slow axis orientations of the second QWP 14058. Can be selected to be). In response to the reflection by the fixed mirror 14062, the handedness of circularly polarized light is reversed. As a second pass, in response to passing through the second QWP, the S-polarized circularly polarized light is converted (temporarily) to P-polarized light, which then passes through polarization selective reflector 14038. To do.

  After passing through the polarization selective reflector 14038, the light passes through the third QWP 14064 and the fourth relay lens element 14066 and is directed to the foveal tracking mirror 14068. In system 14000, image scanning mirror 14030, fixed mirror 14060, and foveal tracking mirror 14068 are spaced from relay lens elements 14032, 14066, 14060 by focal lengths of relay lens elements 14032, 14066, 14060, respectively, and QWP 14040, 14058. , 14064 are positioned after the relay lens elements 14032, 14042, 14060, 14066, so the angle of light incident on the QWPs 14040, 14058, 14064 is relatively low, which is an improvement of the QWPs 14040, 14058, 14064. It leads to performance. According to an alternative embodiment, rather than having a single foveal tracking mirror 1268 that tracks the two angular degrees of freedom of eye movement (eg, azimuth and elevation), the fixed mirror 14062 has a second foveal tracking. A mirror (not shown) can be substituted, one of the two foveal tracking mirrors can be used to track one degree of freedom of eye movement, and a second fovea can be used. The tracking mirror can be used to track the second degree of freedom of eye movement. In such an alternative, a single degree of freedom foveal tracking mirror may be used. Referring again to FIG. 36A, the third relay lens element 14060, in combination with the fourth relay lens element 14066, forms an afocal afocal relay. Foveal tracking mirror 14068 adds deflection of light beam 14028 produced by image scanning mirror 14030 to track the fovea (not shown) of user's eye 210, thereby causing image scanning mirror 14030 to The average angle over the solid angle range of the produced beam angle can be deflected off-axis. The eye tracking camera 14098 tracks the line of sight of the user's eye 210. Eye tracking camera 14098 is coupled to fovea tracking control system 14097. The eye-tracking camera 14098 outputs information, which indicates the line of sight of the eye, which is input to the foveal tracking control system 14097. The foveal tracking control system 14097 is drivingly coupled to the foveal tracking mirror 14068. Based on the eye gaze information received from the eye tracking camera 14098, the foveal tracking control system 14097 directs the foveal tracking mirror 14068 to track the signal to track the fovea 14099 of the user's eye. Output to the mirror 14068. The foveal tracking control system 14097 can use image processing to determine the eye gaze of the user, generate a signal, and control the foveal tracking mirror based on the eye gaze.

  After being reflected by the foveal tracking mirror 14068, the light passes back through the fourth relay lens element 14066 and the third QWP 14064. The first pass of the light through the third QWP 14064 converts the light into circularly polarized light, and the reflection by the foveal tracking mirror 14068 reverses the handedness of the circularly polarized light, resulting in a second pass through the third QWP 14064. Passes the light back into the S-polarized state. The light is now S-polarized so that it is reflected by the polarization selective reflector 14038 and is deflected nominally 90 degrees towards the first QWP 14040. The first QWP 14040 converts S-polarized light into left circularly polarized light (LHCP). The light then passes through the second relay lens element 14042. The fourth relay lens element 14066 is combined with the second relay lens element 14042 to form an afocal afocal compound lens. The relay lens elements 14032, 14042, 14060, 14066 are symmetrically installed at 90-degree intervals around the center of the polarization selective mirror 14038. The generally continuous (in the order of light propagation) relay lens elements 14032, 14042, 14060, 14066 form an afocal afocal relay system. The continuous relay lens elements, positioned to be confocal, share a common focus in the middle across PBS 14036. The relay lens elements 14032, 14042, 14060, 14066 include, by way of non-limiting example, compound lenses including aspheric lenses, aplanatic lenses, hybrid refractive and diffractive lenses and achromatic lenses, eg, refractive lenses with diffractive lenses. Can be included. As used in this description, "relay lens element" includes a single lens or a compound lens.

For LHCP light, the first group of geometric phase lenses 14050 has a positive optical power, which increases the optical power of the first group 14046. For LHCP, the focal length of the first group 14044 is equal to the distance from the principal plane of the first group 14044 to the point F _LHCP . Upon passing through the first group of geometric phase lenses 14050, the LHCP light is converted to RHCP light. Subsequently, the light passes through the second group 14052. For RHCP light, the second group of geometric phase lenses 14056 is such that the positive power of the second group 14052 will be lower than the power of the second group of positive refractive lenses 14054 alone. It has a negative refractive power. Respect RHCP light, the second group 14052 has a focal length equal to the distance from the principal plane of the second group 14052 to point _{F LHCP.} Therefore, for LHCP light entering the 2x afocal magnifying lens 14048, the 2x afocal magnifying lens 14048 acts as a demagnifying lens with a magnifying power of less than 1. Thus, the solid angle range in the direction of the light beam produced by the image scanning mirror 14030, which is deflected by the foveal tracking mirror 14068, is a reduced angle that tracks the fovea of the user as the user's line of sight is shifted. Unexpanded to cover the area. Recall that for incident RHCP, the 2X afocal magnifying lens 14048 has a magnifying power greater than one. A magnification of greater than 1 is used to provide a wider field of view corresponding to the portion of the user's retina outside the fovea.

  In certain embodiments, the second group 14052 is a mirror image of the first group 14046, where the first group of geometric phase lenses 14050 and the second group of geometric phase lenses 14056 are , And the first group of positive refractive lenses 14044 and the second group of positive refractive lenses 14054 are the same. If the refractive lenses 14044, 14054 have surfaces of different refractive power, they are positioned such that surfaces of the same refractive power face each other to maintain the mirror image symmetry of the 2X afocal magnifying lens 14048. obtain. In this case, each group 14046, 14052 may have two different principal planes, depending on whether the geometric phase lens 14050, 14056 acts as a positive or negative lens, but nevertheless The two groups 14046, 14052 have a confocal relationship between the two groups 14046, 14052 to maintain the afocal magnification of the magnifying lens 14048 regardless of whether LHCP or RHCP light is incident on the magnifying lens 14048. They may be separated from each other by a fixed distance that they maintain.

  There are two sets of three augmented reality glasses eyepiece waveguides, including a first eyepiece waveguide 14070, a second eyepiece waveguide 14072, and a third eyepiece waveguide 14074. Positioned beyond a second group 14052 of magnification afocal magnifying lenses 14048 and optically coupled thereto (through free space, as shown). Although three eyepiece waveguides 14070, 14072, 14074 are shown arranged in an overlapping relationship, as an alternative, a different number of eyepiece waveguides are also provided. For example, multiple sets of three eyepiece waveguides may be provided, each set configured to impart a different wavefront curvature (corresponding to a different virtual image distance) to the outgoing light. The three eyepiece waveguides 14070, 14072, 14074 each include three optical incoupling elements 14076, a second optical incoupling element 14078, and a third optical incoupling element 14080. Optical incoupling elements 14076, 14078, 14080 are provided. Each of the three eyepiece waveguides 14070, 14072, 14074 can be configured to transmit light within a particular color channel, eg, red, green, or blue light. In addition, the inner coupling elements 14076, 14078, 14080 are each wavelength selective so as to couple only the light in one color channel into its associated eyepiece waveguide 14070, 14072, 14074. be able to. The internal coupling elements 14076, 14078, 14080 can comprise, for example, a spectrally selective reflective diffraction grating, such as a diffraction grating made from a cholesteric liquid crystal material. Such cholesteric liquid crystal materials have a certain helix pitch, which determines the spectral reflectance band. The internal coupling elements may each comprise, for example, two overlapping layers of cholesteric liquid crystal material, one reflecting LHCP light and the other reflecting RHCP light. Gratings generally have a certain profile pitch, which determines the light deflection angle. When the internal coupling elements 14076, 14078, 14080 are implemented as diffraction gratings, the grating profile pitch of each grating is the critical angle for total internal reflection with respect to the eyepiece waveguide 14070, 14072, 14074 with which the light is associated. Is preferably selected in light of the wavelength of the associated light to be internally coupled so that it is diffracted to an angle greater than. The first, second, and third eyepiece waveguides 14070, 14072, 14074 each include a first exit pupil expander (EPE) 14082, a second EPE 14084, and a third EPE 14086. EPEs 14082, 14084, 14086 may be implemented as transmissive and / or reflective diffraction gratings. The EPEs 14082, 14084, 14086 guide the light out of the waveguides 14070, 14072, 14074 over a relatively wide area as compared to the lateral extent of the internal coupling elements 14076, 14078, 14080. Light propagating in 14070, 14072, 14074 is progressively coupled out of the waveguides 14070, 14072, 14074. An orthogonal pupil expander (OPE), invisible in FIG. 36A, is also provided on the eyepiece waveguides 14070, 14072, 14074 and can be located behind the EPEs 14082, 14084, 14086. The OPE serves to deflect light from the internal coupling elements 14076, 14078, 14080 propagating in the eyepiece waveguides 14070, 14072, 14074 towards the EPEs 14082, 14084, 14086. The OPE may be located in the path of light emanating from the inner coupling elements 14076, 14078, 14080 and the EPE 14082, 14084, 14086 may be outside the path of light emanating from the inner coupling elements 14076, 14078, 14080. However, the OPE may deflect light from the internal coupling elements 14076, 14078, 14080 towards the EPEs 14082, 14084.

  According to an alternative embodiment, the first relay lens element 14032 has a longer focal length than the second 14042, third 14060, and fourth 14066 relay lens elements, a distance equal to the longer focal length, It is spaced from the center of PBS 14036 (taking into account the refractive index of PBS 14036). In this case, the longer focal length first relay lens element 14032, in combination with the second relay lens 14042, imparts an angular magnifying power of greater than 1: 1 to the non-foveal tracking light and a longer focal length first relay lens element 14032. The first relay lens element 14032, in combination with the third relay lens element 14060, imparts an angular magnification of greater than 1: 1 to the foveal tracking light. Recall that the 2x afocal magnifying lens 14048 will demagnify the foveal tracking light and magnify the non-foveal tracking light. Therefore, varying the focal length of the first relay lens element 14032 can be used to set the magnification achieved within the system 14000 without disturbing the symmetry of the design of the 2X afocal magnifying lens 14048. Provides another degree of design freedom that can be achieved. Introducing asymmetry into the design of the 2x afocal magnifying lens 14048 is another possible alternative.

  According to alternative embodiments, other types of dual state lenses are used instead of the geometric phase lenses 14050, 14056. According to one alternative, actively driven electrowetting liquid lenses may be used. A liquid crystal, whose normal axis is aligned with the normal axis and whose extraordinary axis exhibits lens power for light polarized parallel to, made of a material, aligned with a specific direction of the upper layer of diffractive optics. Other alternative lenses may be used, including. In the latter case, the first QWP 14040 can be eliminated because the anisotropic performance of the lens will depend on the linear polarization difference between the foveal and non-foveal tracking light.

  Each orientation of image scanning mirror 14030 corresponds to an angular coordinate in image space when polarization rotation switch 14034 is configured to transmit non-foveal tracking P-polarized light. When the polarization rotation switch 14034 is configured to output foveal-tracked, S-polarized light, the orientation of the image scanning mirror 14030 is combined with the orientation of the foveal tracking mirror 14068 to obtain angular coordinates in image space. To decide. The angle of light beam propagation determined by the orientation of the image scanning mirror and the fovea tracking mirror 14068 is optionally dependent on the magnification of the 2x afocal magnifying lens 14048 and, optionally, the relative focus of the relay lenses 14032, 14042, 14060, 14066. It is multiplied by the magnification factor determined by the distance. The effective size of a pixel defined in the angular image space is proportional to the modulation rate of the laser diodes 14002, 14010, 14016 and the reciprocal of the angular velocity of the movement of the image scanning mirror 14030. To the extent that the movement of the image scanning mirror 14030 can be sinusoidal, the modulation rate of the laser diodes 14002, 14010, 14016 is made inversely proportional to the angular velocity of the image scanning mirror 14030 to reduce or eliminate pixel size variations. May be. When both foveal and non-foveal tracking regions are generated for laser diodes 14002, 14010, 14016, the full potential modulation factor of laser diodes 14002, 14010, 14010 (characteristics of available lasers). Can be used (at least with respect to some point in the field of view), but the full angular range of the image scanning mirror is produced for the foveal tracking region, relative to the relatively small solid angle range. The resolution of the image can be used so that the resolution image can be higher (smaller pixel size) than the resolution of the image produced for a wider field of view.

  In an augmented reality system, in which system 14000 is used, according to an embodiment, virtual content is superimposed on the real world, which is visible to the user through eyepiece waveguides 14070, 14072, 14074. Virtual content is defined as a 3D model (eg, inanimate objects, people, animals, robots, etc.). The 3D model is located and oriented within the 3D coordinate system. In augmented reality systems, for example, through the provision of inertial measurement units (IMUs) and / or visual odometry, the aforementioned 3D coordinate system is aligned with the real world environment (inertia frame of reference) of the user of the augmented reality system. Maintained. The game engine uses its position and orientation to render the left and right eye images of the 3D model for output to the user via system 14000 (and a similar system for the user's other eye). To process the 3D model. A left-eye image, as long as the 3D model is defined in a coordinate system that is fixed to the user's environment, and as long as the user can move and turn its head (carrying augmented reality glasses) in the environment. And the rendering of the right eye image is updated to take into account user head movements and turns. Thus, for example, if a virtual book is displayed stationary on a real table and the user rotates their head 10 degrees to the left, information about the rotation from the IMU or visual odometry subsystem (not shown). In response, the game engine updates the left and right images so that the book appears to maintain its position regardless of the user's head rotation, and the virtual book output by the system 14000. The image will be shifted 10 degrees to the right. In this case, an image for a wider portion of the retina that extends beyond the fovea and an image for a more limited portion of the retina, including the fovea, using polarization rotation switch 14034. , Time multiplexed through system 14000. The image is generated and output by the game engine in synchronization with the operation of the polarization rotation switch 14034. As mentioned above, the game engine produces a left eye image and a right eye image. The game engine also produces narrower FOV left and right fovea images, which causes polarization rotation switch 14034 to output fovea-tracked S-polarization using fovea tracking mirror 14068. Output when configured to. As discussed above, such foveal tracking images are converted to LHCP light and demagnified by a 2x afocal magnifying lens 14048. Such demagnification limits the angular range to a narrow range that includes the fovea (or at least a portion thereof). Demagnification reduces the pixel size, thereby increasing the angular resolution for foveal tracking images.

  37A is a schematic illustration of a 2X afocal magnifying lens 14048 used in the augmented reality eyepiece display system shown in FIG. 36A, according to one embodiment.

  FIG. 37B is a schematic illustration of a bifocal magnifying afocal magnifying lens 15000 that may be used in the augmented reality eyepiece display system 14000 shown in FIG. 36A instead of the afocal magnifying lens 14048, according to another embodiment. The afocal magnifying lens 15000 includes a lens group 15002 that includes a positive refractive lens 15004 and a first geometric phase lens 15006. The afocal magnifying lens 15000 further includes a second geometric phase lens 15008 spaced a distance from the first lens group 15002. The first geometric phase lens 15006 and the second geometric phase lens 15008 have antipodal properties. For light having a handedness that matches the handedness of the geometric phase lens, the geometrical phase lens acts as a positive lens and for light having a handedness opposite to that of the geometrical phase lens, The geometric phase lens acts as a negative lens. In addition, in response to propagating through the geometric phase lens, the handedness of the light is reversed. Therefore, when the first geometric phase lens 15006 acts as a positive lens, the second geometric phase lens 15008 also acts as a positive lens, and the first geometric phase lens 15006 becomes Acting as a negative lens, the second geometric phase lens 15008 will also act as a negative lens. If the first geometric phase lens 15006 acts as a negative lens, the lens group 15002 will have a longer focal length than that of the positive refractive lens 15004 alone. If the first geometric phase lens 15006 acts as a positive lens, the lens group 15002 will have a focal length that is shorter than the focal length of the positive refractive lens 15004 alone.

  In the augmented reality eyepiece display system 14000 shown in FIG. 36A, the P-polarized light output by polarization switch 14034 passes directly through PBS 14036, is not foveated, and is converted to RHCP light by first QWP 14040. Recall, on the other hand, the S-polarized light output from the polarization rotation switch 14034 is reflected by the foveal tracking mirror 14068 and ultimately routed for conversion to LHCP light.

The embodiment shown in FIG. 37B is further described using the assumption that the first geometric phase lens 15006 is counterclockwise and the second geometric phase lens 15008 is clockwise. Will Further, as in the system 14000 shown in FIG. 36A, the LHCP light is fovea-tracked and the RHCP is non-fovea-tracked light, which is the image for a wider FOV (larger portion of the retina). It is assumed to carry modulated light for each. For LHCP light, the first geometric phase lens 15006 acts as a positive lens and the lens group 15002 has a relatively short focal length, which corresponds to the distance from the lens group 15002 to the focus F _LHCP . In transmitting light, the first geometric phase lens 15006 converts LHCP light into RHCP light, in which the second geometric phase lens 15008 has a positive refractive power and a second geometric phase lens 15008. Has a focal length equal to the distance from the optical phase lens 15008 to the point F _LHCP . In this case, the afocal magnifying lens 15000 forms a Keplerian afocal magnifying lens. By proper selection of the focal length of the positive refractive lens 15004 (as will be described further below), the first geometric phase lens 15006 and the second geometric phase lens 15008 are: The magnification of the afocal magnifying lens 15000 in the Keplerian configuration can be chosen to be about 1: 1 or another desired value. For example, assuming image scanning mirror 14030 has an optical angle in the scan range of +/- 10 degrees, such a range of angles may substantially cover the foveal region of the retina.

For RHCP light entering the afocal magnifying lens 15000, the first geometric phase lens 15006 has a negative refractive power and the lens group 15002 corresponds to the distance from the lens group 15002 to the point F _RHCP . It has a relatively longer focal length. The first geometric phase lens 15006 converts the RHCP light into LHCP light, in which respect the second geometric phase lens 15008 measures the distance from the second geometric phase lens 15008 to the point F _RHCP. With a negative focal length. In this case, the afocal magnifying lens 15000 can be configured as a Galilean afocal magnifying lens and have a magnifying power substantially greater than 1: 1 eg 3: 1. Therefore, the RHCP light that is incident on the afocal magnifying lens (which is not foveal tracked) causes the image-modulated light to reach a larger portion of the retina beyond the fovea (compared to the area illuminated by the LHCP light). Can be provided. Note that the systems 14000, 15000 can be configured to reverse the role of RHCP and LHCP light.

For a given focal length of the positive refracting lens 15004 and a given magnitude of the focal length of the first geometric phase lens 15004, the lens group 15002 has the palmity of the incident light (as described above). ), According to), one of two focal lengths equal to the distance from the lens group 15002 to the points F _LHCP and F _RHCP . The second geometric phase lens 15008 should be centered on the midpoint between F _LHCP and F _RHCP, and the focal length of the second geometric phase lens 15008 is F _LHCP and F _LHCP It should be set approximately in the middle of the distance to _RHCP . The magnification factor of the Keplerian configuration is approximately equal to the negative ratio of the distance from the lens group 15002 to the point F _LHCP divided by the distance from the point F _LHCP to the second geometric phase lens 15008. The magnification of the Galilean configuration is approximately equal to the ratio of the distance from the lens group 15002 to the point F _RHCP divided by the distance from the second geometric phase lens 15008 to the point F _RHCP .

  The 2X afocal magnifying lens 14048, 15000 can also be used in other types of optical devices, including, by way of non-limiting example, telescopes, binoculars, cameras, and microscopes. In a system where a real image is to be formed, afocal magnifying lenses s14048, 15000 can be used in combination with additional optical elements (eg lenses, convex mirrors).

  Referring to FIG. 36A, according to an alternative embodiment, the fixed mirror 14062 is replaced with a second image scanning mirror and includes a laser diode, a collimating lens, and an RGB dichroic combination cube (FIG. 36A). 36A) can be used to provide RGB image modulated light to the second scanning mirror. The second subsystem and the second scanning mirror would be dedicated to provide foveal tracking light. In this case, the polarization rotation switch 14034 and the second QWP 14058 can be omitted and both foveal and non-foveal tracking light can be produced at the same time. In all such alternatives, the laser diode would be oriented to inject P-polarized light into PBS 14036.

  FIG. 36B schematically illustrates another augmented reality eyepiece display system 14000B. The following description of the embodiment shown in FIG. 36B will focus on the differences in that system 14000B has certain aspects in common with system 14000 shown in FIG. 36A. In system 14000B, the 2x afocal magnifying lens 14048, the second QWP 14058, and the polarization rotation switch 14034 are eliminated. A longer focal length first relay lens element 14032B is used such that the combination of the first relay lens element 14032B and the second relay lens element 14042 increases the angular field of view of the light scanned by the scanning mirror 14030. To be done. Scanning mirror 14030 is used to cover the full field of view of system 14000B minus the high resolution foveal tracking of the FOV. The second scanning mirror 14030 can be installed at a distance from the first relay lens element 14032B that is equal to the focal length of the first relay lens element 14032B. The first RGB light engine 14095 is configured to output P-polarized light, and in the absence of the polarization rotation switch 14034, the light scanned by the scanning mirror 14030 is the first relay lens element 14032B and the second relay lens element 14032B. It will be coupled through the relay lens element 14042.

  The fixed mirror 14062 used in the system 14000 (FIG. 36A) replaces the second scanning mirror 14030B. A second primary color (eg, red-blue-green (RGB)) light engine 14096 complements a first primary color (eg, red-blue-green (RGB)) 14095. The second RGB light engine 14095 is coupled to a second scanning mirror 14030B through a collimating lens 14004B, 14012B, 14018B and a second RGB dichroic combiner cube 14008B to a second red, green, and blue laser diode 14002B. , 14010B, 14016B. The additional elements of the second RGB light engine 14096 correspond to the elements of the first RGB light engine 14095 described above and have a common numerical part and an additional subscript'B ', see Be labeled with a number. The P-polarized light output by the second RGB light engine 14096 and angularly scanned by the second scanning mirror 14030 is infinite formed by the third relay lens element 14060 and the fourth relay lens element 14066. It is optically coupled to the foveal tracking mirror 14068 through a focus relay system and passes through a third QWP 14064 as it reaches the foveal tracking mirror. In response to being angularly displaced by the foveal tracking mirror 14068, light is retroreflected through the fourth relay lens element 14066 and the third QWP 14068, where its polarization state changes to S-polarization. And is reflected by the polarization-selective mirror towards the first QWP 14040 and the second relay lens element 14042 and then impinges on the inner coupling elements 14076, 14078, 14080. It should be appreciated that the first and second RGB light engines 14095, 14096 may utilize primary colors other than or in addition to red, blue, and green.

The augmented reality eyepiece display system 14000B is capable of simultaneously outputting a foveal tracking high resolution image and a non-foveal tracking wider field of view image. By avoiding the need to time multiplex higher resolution foveal tracking images and wider field-of-view images (as is the case for the system shown in FIG. 36A), system 14000B makes higher frame rates easier. Can be achieved.

V. Tracking the entire visual field with eye gaze

  According to some embodiments, as shown in FIGS. 26E-26F, instead of presenting the first image stream in a static position, both the first image stream and the second image stream may be Can be dynamically shifted according to the current fixation point of. 38A-38B schematically illustrate exemplary configurations of images that may be presented to a user, according to some embodiments. FIG. 38A illustrates how the second image stream 16020 can be substantially centered in the first image stream 16010. In some embodiments, it may be desirable to offset the second image stream 16020 from the center of the first image stream. For example, it may be desirable to offset the second image stream 16020 toward the nose side of the first image stream because the user's field of view extends further in the temple direction than in the nose direction. During operation, the first and second image streams are continuously shifted according to the user's current fixation point as determined in real time using eye gaze tracking techniques, as shown in FIG. 38B. Can be done. That is, the first image stream 16010 and the second image stream 16020 can be interlockedly shifted such that the user is typically looking directly at the center of both image streams. The grid squares in FIGS. 38A-38B graphically depict image points, such as fields of view 3002, 3004, and 3006 as defined in two-dimensional angular space, as described above with reference to FIG. Note that it represents.

  Similar to the embodiment depicted in FIGS. 26A-26B, the second image stream 16020 represents a high resolution image stream having a relatively narrow FOV that may be displayed within the boundaries of the first image stream 16010. In some embodiments, the second image stream 16020 is dynamically adjusted in real-time to an angular position that matches the user's current fixation point based on data obtained using eye gaze tracking techniques. One or more images of virtual content that may be captured by a second, different virtual camera having an orientation that is in the rendering space may be represented. In these examples, the high resolution second image stream 16020 would be captured by a foveal tracking virtual camera, such as the foveal tracking virtual camera described above with reference to FIGS. 26A-26D. One or more images of different virtual content can be represented. In other words, the viewpoint in rendering space at which one or more images of the virtual content represented by the second image stream 16020 is captured is that the viewpoint associated with the second image stream 5020E is the fovea of the user. It can be reoriented as the eye gaze of the user changes so as to be permanently aligned with the eye.

  For example, the second image stream 16020 contains virtual content located within the first region of the rendering space when the user's gaze is fixed in the first position as illustrated in FIG. 38A. be able to. As the user's line of sight moves to a second position that is different from the first position, the viewpoint associated with the second image stream 16020 is such that the second image stream 16020 is as illustrated in FIG. 38B. , Can be adjusted to include virtual content located within the second region of the rendering space. In some embodiments, the first image stream 16010 has a wide FOV, but low angular resolution, as indicated by the sparse grid. The second image stream 16020 is a narrow FOV, but with high angular resolution, as shown by the dense grid.

  39A-39B illustrate some of the principles described in FIGS. 38A-38B using some example images that may be presented to a user, according to some embodiments. In some embodiments, one or more of the images and / or image streams depicted in FIGS. 39A-39B may include one or more of the depth planes described above with reference to FIG. 25B. May represent a two-dimensional image or part thereof to be displayed in a particular depth plane, such as That is, such images and / or image streams may represent 3-D virtual content projected onto at least one two-dimensional surface at a fixed distance from the user. In such an embodiment, such images and / or image streams may include one or more lights with an angular field of view similar to those described above with reference to FIGS. 26A-26D and 28A-28B. It should be appreciated that it may be presented to the user as a field.

  As depicted, the content of the first image stream 17010 comprises a portion of a tree. During the first time period represented by FIG. 39A, the eye tracking sensor focuses the user's line of sight (ie, foveal vision) on the first region 17010-1 within the viewable region 17000. You can decide that. In this example, the first area 17010-1 includes the branches of the lower tree. The second image stream 17020 may be located within the first region 17010-1 and have a higher resolution than the first image stream. The first and second image streams can be displayed in parallel or at high speed in succession at positions determined to correspond to the user's current eye gaze.

  During the second time period represented by FIG. 39B, it has been detected that the user's line of sight has shifted to a second region 17010-2 within the viewable region 1500, corresponding to the upper tree branch. You can As depicted, during the second time period, the position and content of the first and second image streams change to correspond to the second region 17010-2. The content of both the first image stream 17010 and the second image stream 17020 can include a second region 17010-2 of the tree. The first and second image streams can be displayed in parallel or at high speed in succession. The further detected movement of the user's line of sight can be adapted in the same manner so that both the first and second image streams remain aligned with the user's current line of sight.

  Similar to the embodiment illustrated in FIGS. 28C-28D, the higher resolution second image stream 17020 overlays a portion of the first image stream 17010 within the fovea view of the user, thus lower resolution. The first image stream 17010 cannot be perceived or perceived by the user. In addition, the first image stream 17010, which has a wide field of view, may encompass a substantial portion of the user's vision, thus preventing the user from completely perceiving the boundaries of the light field display. Therefore, the present technique may provide a more immersive experience to the user.

  40A-40D schematically illustrate a display system 18000 for projecting an image onto a user's eye, according to some embodiments. Display system 18000 includes an image source 18010. The image source 18010 can be configured to project a first light beam 18052 associated with the first image stream and a second light beam 18054 associated with the second image stream. The first image stream can be a wide FOV and low resolution image stream and the second image stream can be a narrow FOV and high resolution image stream, as discussed above with reference to FIGS. 38A-38B. Can be In some embodiments, the first light beam 18052 and the second light beam 18054 can be time division multiplexed, polarization division multiplexed, wavelength division multiplexed, or the like.

  The display system 18000 can further include a 2D scanning mirror 18020 configured to reflect the first light beam 18052 and the second light beam 18054. In some embodiments, the 2D scanning mirror 18020 allows the first light beam 18052 and the second light beam 18054 to respectively focus the first image stream and the second image stream into a user's foveal view. As can be projected, it can be tilted in two directions based on the fixation position of the user's eye.

  Display system 18000 can further include switchable optical element 18040. Switchable optical element 18040 is shown as a single element, but can include a pair of sub-switchable optical elements that function as a switchable relay lens assembly. Each sub-switchable optical element is switched to a first state to operate as an optical lens with a first optical power, as illustrated in FIGS. 40A and 40C, or illustrated in FIGS. 40B and 40D. Thus, the second state can be switched to operate as an optical lens with a second optical power different from the first optical power. Each sub-switchable optical element can be, for example, a liquid crystal variable focus lens, a tunable diffractive lens, a deformable lens, or a multifocal birefringent lens, according to various embodiments.

When the first light beam 18052 and the second light beam 18054 are time division multiplexed, the switchable optical element 18040 and the scan mirror 18020 can operate as follows. It is assumed that the user's gaze is fixed at the first position during the first time period. Scanning mirror 18020 is configured to direct first light beam 18052 and second light beam 18054 toward a first position during a first period of time, as illustrated in FIGS. 40A and 40B. And in a first orientation. During the first time slot of the first time period (stage A ₁ ), at which the image source 18010 outputs the first light beam 18052, the switchable optical element 18040 is switched to the first as shown in FIG. 40A. It can be switched to a first state, acting as an optical lens with a refractive power of 1. During the second time slot (stage A ₂ ) of the first time period, when the image source 18010 outputs the second light beam 18054, the switchable optical element 18040 is switched to the first time period as shown in FIG. 40B. It can be switched to a second state, acting as an optical lens with a refractive power of 2. Thus, the first light beam 18052 may present a first image stream with a wider FOV than the second image stream presented by the second light beam 18054. The second light beam 18054 is angularly expanded so that it can.

Here, it is assumed that the line of sight of the user has moved from the first position to the second position during the second time period. Scanning mirror 18020 is configured to direct first light beam 18052 and second light beam 18054 toward a second position during a second period of time, as illustrated in FIGS. 40C and 40D. And in a second orientation. During the first time slot of the second time period (stage B ₁ ) when the image source 18010 outputs the first light beam 18052, the switchable optical element 18040 is switched to the _first time slot as shown in FIG. 40C. It can be switched to a first state, acting as an optical lens with a refractive power of 1. Image source 18010 outputs a second light beam 18 054, between the second second time slot of the time period (Step _{B 2),} switchable optical element 18040, as illustrated in Figure 40D, the It can be switched to a second state, acting as an optical lens with a refractive power of 2.

  When the first light beam 18052 and the second light beam 18054 are polarization division multiplexed, the switchable optical element 18040 is for the first light beam 18052 as illustrated in FIGS. 40A and 40C. Acting as an optical lens with a first optical power, and as a optical lens with a second optical power for a second light beam 18054, as shown in FIGS. 40B and 40D, is multifocal. A birefringent lens can be included.

  When the first light beam 18052 and the second light beam 18054 are wavelength division multiplexed, the switchable optical element 18040 is for the first light beam 18052 as illustrated in FIGS. 40A and 40C. Wavelength dependent to act as an optical lens with a first optical power, and as an optical lens with a second optical power for a second light beam 18054, as illustrated in FIGS. 40B and 40D. A multifocal lens can be provided.

  41A-41D schematically illustrate a display system 19000 for projecting an image onto a user's eye according to some other embodiments. Display system 19000 can be similar to display system 18000, but switchable optical element 18040 can be disposed on the surface of scan mirror 18020. For example, switchable optical element 18040 can be one or more substrates layered on the surface of scan mirror 18020.

  In some further embodiments, switchable optical element 18040 can be located elsewhere within display system 19000. For example, it can be positioned between the image source 18010 and the scanning mirror 18020.

  In some other embodiments, a polarizing beamsplitter or dichroic beamsplitter may be used to demultiplex the first light beam 18052 and the second light beam 18054 into two separate optical paths. Although, both optical paths intersect the reflective surface of scan mirror 18020.

In other embodiments, more than two image streams may be presented to the user such that the transition in resolution from the user's fixation point to the user's peripheral vision is more gradual in appearance. For example, a third image stream having a medium FOV and medium resolution can be presented in addition to the first image stream and the second image stream. In such cases, additional relay lens assemblies and / or scanning mirrors can be utilized to provide additional optical paths for additional image streams.

VI. Time multiplexing scheme

  In some embodiments, the high FOV low resolution image stream (ie, the first image stream) and the low FOV high resolution image stream (ie, the second image stream) can be time division multiplexed.

  FIG. 42 shows a graph illustrating an exemplary time division multiplexing pattern suitable for use with the high FOV low resolution image stream and the low FOV high resolution image stream. As shown, the high FOV low resolution image stream and the low FOV high resolution image stream are distributed in alternating time slots. For example, each time slot may be approximately 1 / 85th of a second in duration. Thus, the high FOV low resolution image stream and the low FOV high resolution image stream may each have a refresh rate of about 42.5 Hz. In some embodiments, the angular region corresponding to the light field of the low FOV high resolution image stream overlaps a portion of the angular region of the light field corresponding to the high FOV low resolution image stream, and in the overlapped angular region. Make the effective refresh rate about 85 Hz (ie, twice the refresh rate of each individual image stream).

  In some other embodiments, the time slots for the high FOV low resolution image stream and the time slots for the low FOV high resolution image stream can have different durations. For example, each time slot for a high FOV low resolution image stream can have a duration greater than 1 / 85th of a second, and each time slot for a low FOV high resolution image stream can be a 1 / 85th of a second. It can have a shorter duration, or vice versa.

  FIG. 43 schematically illustrates a display system 21000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 21000 may share some elements in common with display system 8000 as illustrated in FIGS. 30A-30B. For this reason, the discussion of those common elements associated with Figures 30A-30B is equally applicable here. Image source 21002 can be configured to simultaneously provide a high FOV low resolution image stream in a first polarization state and a low FOV high resolution image stream in a second polarization state. For example, the first polarization state can be linear polarization in a first direction and the second polarization state can be linear polarization in a second direction orthogonal to the first direction, or Alternatively, the first polarization state can be left-hand circular polarization and the second polarization state can be right-hand circular polarization. Similar to the display system 8000 illustrated in FIGS. 30A-30B, the display system 21000 includes a high FOV low that propagates a light beam projected by an image source (eg, image source 21002) along a first optical path. A polarizing beam splitter 21004 for separating a first light beam associated with the resolution image stream and a second light beam associated with the low FOV high resolution image stream propagating along the second optical path. Including.

  Similar to the display system illustrated in FIGS. 30A-30B, display system 21000 includes a first optical lens (lens A) positioned between image source 21002 and beamsplitter 21004, and a first optical path. A second optical lens (lens B) positioned downstream of the beam splitter 21004 along and a third optical lens (lens C) positioned downstream of the beam splitter 21004 along a second optical path. Can be included. In some embodiments, the combination of the first optical lens (lens A) and the second optical lens (lens B) is greater than 1, as described above with respect to FIGS. 30A-30B and 31A-31B. , A combination of the first optical lens (lens A) and the third optical lens (lens C) can provide an angular magnification for the first light beam and is substantially equal to or 1 An angular magnification factor for the second light beam that is less than can be provided. Therefore, the first light beam can project an image stream having a wider FOV than that projected by the second light beam.

  Similar to the display system 8000 illustrated in FIGS. 30A-30B, the display system 21000 also fixes the eye of the user to dynamically project a second light beam associated with the low FOV high resolution image stream. Includes a fovea tracker 21006, which may take the form of a scanning mirror (eg, a MEMS mirror), which may be controlled based on viewing position.

  The display system 21000 can also include a first incoupling grating (ICG) 21010 and a second ICG 21020 coupled to the eyepiece 21008. The eyepiece 21008 can be a waveguide plate configured to propagate light therein. Each of the first ICG 21010 and the second ICG 21020 can be a diffractive optical element (DOE) configured to diffract a portion of light incident thereon into an eyepiece 21008. The first ICG 21010 can be positioned along the first optical path to couple a portion of the first light beam associated with the high FOV low resolution image stream into the eyepiece 21008. The second ICG 21020 can be positioned along the second optical path to couple a portion of the second light beam associated with the low FOV high resolution image stream into the eyepiece 21008.

  The display system 21000 can also include a first switchable shutter 21030 and a second switchable shutter 21040. The first switchable shutter 21030 is positioned along a first optical path between the second optical lens (lens B) and the first ICG 21010. The second switchable shutter 21040 is positioned along a second optical path between the fovea tracker and the second ICG 21020. The operation of the first switchable shutter 21030 and the second switchable shutter 21040 is such that the high FOV low resolution image stream and the low FOV high resolution image stream are time-division multiplexed sequences (eg, as illustrated in FIG. 42). ), They can be synchronized with each other so that they are time division multiplexed according to The first switchable shutter 21030 is open for a period of time corresponding to the first time slot associated with the high FOV low resolution image and of the second time slot associated with the low FOV high resolution image stream. Can be closed for a while. Similarly, the second switchable shutter 21040 is open during the second time slot and closed during the first time slot.

  Thus, the high FOV low resolution image stream is combined into the eyepiece 21008 using the first ICG 21010 during the first time slot (eg, when the first switchable shutter 21030 is opened). , The low FOV high resolution image stream is combined into an eyepiece 21008 using a second ICG 21020 during a second time slot (eg, when the second switchable shutter 21040 is opened). .. Once the high FOV low resolution image stream and the low FOV high resolution image stream are combined into the eyepiece 21008, they may be guided and outcoupled into the user's eye (eg, an outcoupling grid). By).

  FIG. 44 schematically illustrates a display system 22000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 22000 may share some elements in common with display system 8000 illustrated in FIGS. 30A-30B. The discussion of those elements in relation to Figures 30A-30B is equally applicable here. The high FOV low resolution image stream and the low FOV high resolution image stream provided by image source 22002 can be time division multiplexed and can be in a given polarized state.

  The display system 22000 can include a switchable polarization rotator 22010 (eg, a ferroelectric liquid crystal (FLC) cell with half-wave delay). The operation of the switchable polarization rotator 22010 is such that the switchable polarization rotator 22010 does not rotate the polarization of the high FOV low resolution image stream (or rotates it by a very small amount), but does not rotate the polarization of the low FOV high resolution image stream. A high FOV low resolution image stream and low in time division multiplexing (eg, as shown in FIG. 42), such that it rotates about 90 degrees (ie, introduces a phase shift π) or vice versa. It can be electronically programmed to be synchronized with the frame rate of the FOV high resolution image stream. Therefore, after passing through the switchable polarization rotator 22010, the polarization of the high FOV low resolution image stream may be orthogonal to the polarization of the low FOV high resolution image stream. For example, a high FOV low resolution image stream can be s-polarized, a low FOV high resolution image stream can be p-polarized, or vice versa. In other embodiments, the high FOV low resolution image stream can be left-hand circularly polarized and the low FOV high resolution image stream can be right-handed circularly polarized or vice versa. ..

  The display system 22000 propagates a light beam along a first optical path towards a first ICG 21010, a first light beam associated with a high FOV low resolution image stream, and a second optical path. A polarizing beamsplitter 22004 may be included for splitting into a second light beam associated with a low FOV high resolution image stream that propagates toward the second ICG 21020.

  The display system 22000 also includes a static polarization rotator 22020 positioned along one of the two optical paths, eg, along the second optical path, as illustrated in FIG. You can The static polarization rotator 22020 has a low FOV high resolution image stream and a high resolution image stream so that the two image streams may have substantially the same polarization as they enter the first ICG 21010 and the second ICG 21020, respectively. It can be configured to rotate the polarization of one of the FOV low resolution image streams. This may be advantageous if the first ICG 21010 and the second ICG 21020 are designed to have higher diffraction efficiency for certain polarizations. The static polarization rotator 22020 can be, for example, a half wave plate.

  FIG. 45 schematically illustrates a display system 23000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 23000 may share some elements in common with display system 8000 illustrated in FIGS. 30A-30B. The discussion of those elements in relation to Figures 30A-30B is equally applicable here. Image source 23002 can be configured to provide time-division multiplexed high FOV low resolution image streams and low FOV and high resolution image streams.

  Here, instead of a beam splitter, the display system 23000 includes a switchable reflector 23004. The switchable reflector 23004 can be switched between a reflective mode in which the incident light beam is reflected and a transmissive mode in which the incident light beam is transmitted. Switchable reflectors may include electroactive reflectors with liquid crystals embedded in a substrate host medium such as glass or plastic. Liquid crystals that change their index of refraction as a function of applied current may also be used. Alternatively, lithium niobate may be utilized as the electroactive reflective material instead of the liquid crystal. The operation of the switchable reflector 23004 is such that the switchable reflector 23004 is in reflective mode when a high FOV low resolution image stream arrives and in transmissive mode when a low FOV high resolution image stream arrives. It can be electronically programmed to be synchronized with the frame rate of the high FOV low resolution image stream and the low FOV high resolution image stream in split multiplexing (eg, as shown in FIG. 42). Thus, the high FOV low resolution image stream can be reflected by the switchable reflector 23004 along the first optical path towards the first ICG 21010 and the low FOV high resolution image stream can be switched. The reflector 23004 may allow transmission along the second optical path towards the second ICG 21020.

Alternatively, the switchable reflector 23004 is replaced by a dichroic mirror configured to reflect light within a first set of wavelength ranges and transmit light within a second set of wavelength ranges. You can The image source 23002 can be configured to provide a high FOV low resolution image stream in the first set of wavelength ranges and a low FOV high resolution image stream in the second set of wavelength ranges. For example, the first set of wavelength ranges may correspond to red, green, and blue (RGB) colors, and the second set of wavelength ranges includes RGB in a different hue than that of the first set of wavelength ranges. Can correspond to color. In some embodiments, the high FOV low resolution image stream and the low FOV high resolution image stream are time division multiplexed, eg, as illustrated in FIG. In some other embodiments, the high FOV low resolution image stream and the low FOV high resolution image stream are presented simultaneously.

VII. Polarization multiplexing scheme

  In some embodiments, the high FOV low resolution image stream and the low FOV high resolution image stream can be polarization division multiplexed. The image source provides a first set of RGB lasers for providing a high FOV low resolution image stream in a first polarization and a low FOV high resolution image stream in a second polarization different from the first polarization. Second set of RGB lasers. For example, the high FOV low resolution image stream can be s-polarized, the low FOV high resolution image stream can be p-polarized, or vice versa. Alternatively, the high FOV low resolution image stream can be left-handed circularly polarized and the low FOV high resolution image stream can be right-handed circularly polarized or vice versa.

  FIG. 46 schematically illustrates a display system 25000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 25000 may share some elements in common with display system 8000 illustrated in FIGS. 30A-30B. The discussion of those elements in relation to Figures 30A-30B is equally applicable here. Image source 25002 can be configured to provide polarization division multiplexed high FOV low resolution image streams and low FOV and high resolution image streams, as discussed above.

  The display system 25000 propagates a light beam along a first optical path towards a first ICG 21010, a first light beam associated with a high FOV low resolution image stream, and a second optical path. A polarizing beam splitter 25004 may be included for splitting into a second light beam associated with a low FOV high resolution image stream that propagates along the second ICG 21020.

Display system 25000 also includes a static polarization rotator 25020 positioned along one of the two optical paths, eg, along the second optical path, as illustrated in FIG. You can The static polarization rotator 25020 has a low FOV high resolution image stream and a high resolution so that the two image streams can have substantially the same polarization as they enter the first ICG 21010 and the second ICG 21020, respectively. It can be configured to rotate the polarization of one of the FOV low resolution image streams. This may be advantageous if the first ICG 21010 and the second ICG 21020 are designed to have higher diffraction efficiency for certain polarizations. The static polarization rotator 25020 can be, for example, a half wave plate.

VIII. Optical architecture for internal coupling of images projected into the opposite side of the eyepiece

  In some embodiments, instead of having two ICGs (i.e., having separate pupils) laterally separated from each other, the display system provides a high FOV low resolution image stream and a low FOV high resolution image stream. , Can be configured to be incident on opposite sides of the same ICG (ie, having a single pupil).

  FIG. 47 schematically illustrates a display system 26000 for projecting an image stream onto a user's eye, according to some embodiments. The display system 26000 includes a first image source 26002to configured to provide a high FOV low resolution image stream and a second image source 26004 configured to provide a low FOV high resolution image stream. Can be included.

  Display system 26000 may also include a first optical lens (lens A) and a second optical lens (lens B) positioned along the first optical path of the high FOV low resolution image stream. .. In some embodiments, the combination of the first optical lens and the second optical lens provides an angular magnification of greater than 1 for the first light beam associated with the high FOV low resolution image stream, Thereby, a wider FOV for the first light beam can be provided.

  Display system 26000 also includes an eyepiece 26008 and an incoupling grating (ICG) 26010 coupled to eyepiece 26008. The eyepiece 26008 can be a waveguide plate configured to propagate light therein. The ICG 26010 can be a diffractive optical element configured to diffract a portion of the light incident on it into the eyepiece 26008. As the first light beam associated with the high FOV low resolution image stream is incident on the first surface 26010-1 of the ICG 26010, a portion of the first light beam is reflected in a reflective mode (eg, first order reflection). At, the light is diffracted into the eyepiece 26008, which can then be subsequently propagated through the eyepiece 26008 and outcoupled towards the user's eye.

  The display system 26000 can also include a third optical lens (lens C) and a fourth optical lens (lens D) positioned along the second optical path of the low FOV high resolution image stream. .. In some embodiments, the combination of the third optical lens and the fourth optical lens is substantially equal to or less than 1 for the second light beam associated with the low FOV high resolution image stream. The angular magnification can be provided. Therefore, the second light beam may have a narrower FOV than that of the first light beam.

  The display system 26000 may further be controlled based on the eye fixation position of the user's eye to dynamically project a second light beam associated with a low FOV and high resolution image stream, a scanning mirror (eg, A fovea tracker 26006, such as a MEMS mirror) can be included.

  The second light beam associated with the low FOV high resolution image stream may be incident on the second surface 26010-1 of the ICG 26010 opposite the first surface 26010-2. A portion of the second light beam can be diffracted in transmission mode (eg, first-order transmission) into eyepiece 2408, which is then subsequently propagated through eyepiece 26008 to the user's eye. Can be externally coupled towards the eye.

  As explained above, the display system 26000 uses a single ICG 26010 instead of two separate ICGs, as illustrated in Figures 43-46. This can simplify the design of the eyepiece.

  FIG. 48 schematically illustrates a display system 27000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 27000 may share some elements in common with display system 8000 illustrated in FIGS. 30A-30B. The discussion of those elements in relation to Figures 30A-30B is equally applicable here. The display system 27000 can include an image source 27002 configured to provide a high FOV low resolution image stream and a low FOV and high resolution image stream that are time division multiplexed. In some embodiments, the image source 27002 can take the form of a pico projector.

  The display system 27000 is positioned downstream of the image source 27002 and routes the high FOV low resolution image stream and the low FOV and high resolution image stream from the unpolarized state to the S-polarized and P-polarized or polarized states such as RHCP and LHCP polarized states. A polarizer 27010 may be included that is configured to convert to a closed state.

  Display system 27000 may further include a switchable polarization rotator 27020 positioned downstream of polarizer 27010. The operation of the switchable polarization rotator 27020 is such that the switchable polarization rotator 27020 does not rotate the polarization of the high FOV low resolution image stream (or rotates it by a very small amount) and does not rotate the polarization of the low FOV high resolution image stream. Synchronized with the frame rate of the high FOV low resolution image stream and the low FOV high resolution image stream in time division multiplexing such that it rotates about 90 degrees (ie, introduces a phase shift π) or vice versa. Can be programmed electronically. Thus, after passing through the switchable polarization rotator 27020, the polarization of the high FOV low resolution image stream may be orthogonal to the polarization of the low FOV high resolution image stream. For example, a high FOV low resolution image stream can be s-polarized, a low FOV high resolution image stream can be p-polarized, or vice versa. In other embodiments, the high FOV low resolution image stream can be left-handed circularly polarized and the low FOV high resolution image stream can be right-handed circularly polarized or vice versa. .

  The display system 27000 is further configured to polarize the high FOV low resolution image stream along a first optical path and transmit the low FOV high resolution image stream along a second optical path. Including 27004.

  Display system 27000 further includes a first optical lens (lens A) positioned in front of polarizing beam splitter 27004 and a second optical lens positioned downstream of polarizing beam splitter 27004 along a first optical path. (Lens B) and a third optical lens (lens C) positioned downstream of the beam splitter 27004 along the second optical path. In some embodiments, the combination of the first optical lens (lens A) and the second optical lens (lens B) is greater than 1, as described above with respect to FIGS. 30A-30B and 31A-31C. , A high FOV low resolution image stream can be provided with an angular magnification and the combination of the first optical lens (lens A) and the third optical lens (lens C) is substantially equal to 1 or An angular magnification factor for a low FOV high resolution image stream that is less than 1 can be provided. Thus, the high FOV low resolution image stream may be projected on the user's eye with a wider FOV than that projected by the low FOV high resolution image stream.

  The display system 27000 may further be controlled based on a user's eye fixation position to dynamically project a second light beam associated with a low FOV and high resolution image stream, such as a scanning mirror (eg, A fovea tracker 27006, such as a MEMS mirror) can be included.

  The display system 27000 can further include an eyepiece 27008 and an internal coupling grating (ICG) 27050 coupled to the eyepiece 27008. The eyepiece 27008 can be a waveguide plate configured to propagate light therein. The ICG 27050 can be a diffractive optical element configured to diffract a portion of the light incident on it into the eyepiece 27008.

  The display system 27000 can further include a first reflector 27030 positioned downstream of the second optical lens (Lens B) along the first optical path. The first reflector 27030 can be configured to reflect the high FOV low resolution image stream towards the ICG 27050. As the first light beam associated with the high FOV low resolution image stream is incident on the first surface 27050-1 of the ICG 27050, a portion of the first light beam is transmitted in a transmission mode (eg, primary transmission). At, the light is diffracted into the eyepiece 27008, which subsequently propagates through the eyepiece 27008 and can be outcoupled towards the user's eye.

  Display system 27000 may further include a second reflector 27040 positioned downstream of fovea tracker 27006 along a second optical path. The second reflector 27040 may be configured to reflect the low FOV high resolution image stream towards the ICG 27050. A portion of the second light beam as the second light beam associated with the low FOV high resolution image stream is incident on the second surface 27050-2 of ICG 27050 opposite the first surface 27050-1. Is diffracted into the eyepiece 27008 in a reflective mode (eg, first order reflection), which subsequently propagates through the eyepiece 27008 and can be outcoupled towards the user's eye.

FIG. 49 schematically illustrates a display system 28000 for projecting an image stream onto a user's eye, according to some embodiments. Display system 28000 is similar to display system 27000, but does not include an ICG. Instead, the display system 28000 includes a first internal coupling prism 28030 (instead of the first reflector 27030 in the display system 27000) for coupling the high FOV low resolution image stream into the eyepiece 27008. A second internal coupling prism 28040 (instead of the second reflector 27040 in the display system 27000) for coupling the low FOV high resolution image stream into the eyepiece 27008. The index of refraction of the first incoupling prism 28030 and the index of refraction of the second incoupling prism 28040 are such that a percentage of the optical power contained in the first light beam associated with the high FOV low resolution image stream and a low power. A proportion of the optical power contained in the second light beam associated with the FOV high resolution image stream is reflected in the eyepiece 27008 by the first inner coupling prism 28030 and the second inner coupling prism 28040, respectively. Can be suitably selected for the index of refraction of the eyepiece 27008.

IX. High field of view and high resolution foveated display using overlapping optical paths

  In some embodiments, the display system does not utilize a PBS to separate the composite image stream into two image streams that propagate in different directions, and a high FOV low resolution image stream and a low FOV high resolution image stream are provided. , May be configured to be provided in an eyepiece. Rather, the high FOV low resolution image stream and the low FOV high resolution image stream may take substantially the same path from the image source to the eyepiece, which may eliminate the PBS. This can have the advantage of providing a compact form factor for display systems.

  FIG. 50 schematically illustrates a display system 50000 for projecting an image stream onto a user's eye. Display system 50000 may include an image source 50002 configured to provide modulated light containing image information. In some embodiments, the image source 50002 presents the high FOV low resolution image in a time multiplexed manner, such as by interleaving frames from the first image stream with frames of the second stream. And a second image stream used to present the low FOV high resolution image stream.

  Display system 50000 may also include variable optics 50004. In some embodiments, the variable optics 50004 provides an angular magnification factor for a ray 50030 associated with a high FOV low resolution image stream that is different from a ray 5020 associated with a low FOV high resolution image stream. A high FOV low resolution image stream, thereby enabling the projection of the high FOV low resolution image stream out of the waveguide 50010 and providing a wider FOV than that projected by the low FOV high resolution image stream. It should be appreciated that the range of angles at which the incoupling light is incident on the ICG50006 is preferably preserved in outcoupling that light from the waveguide 50010. Thus, internally coupled light incident on the ICG50006 at a wide range of angles also propagates away from the waveguide 50010 at a wide range of angles when it is outcoupled, thereby resulting in a high FOV and more angular magnification. I will provide a. Conversely, light incident on the ICG50006 at a relatively narrow range of angles also propagates away from the waveguide 50010 at a narrow range of angles when outcoupled, thereby providing a low FOV and Provides low angle magnification.

  In addition, in order to select the appropriate level of angular magnification, the variable optics 50004 associates with the high FOV low resolution image stream so that it has different optical properties than the light associated with the low FOV high resolution image stream. The emitted light may be modified. Preferably, the function of the variable optics 50004 and the light properties of each image stream are matched such that changing the relevant properties of the light changes the refractive power and focal length provided by the variable optics 50004. .. For example, the high FOV low resolution image stream may have a first polarization and the low FOV low resolution image stream may have a second polarization. Preferably, the variable optics 50004 has different optical powers and different due to the different polarizations of the light propagating therethrough such that the desired optical power can be selected by providing light of a particular associated polarization. It is configured to provide a focal length. The first polarized light may be right circularly polarized light (RHCP), left circularly polarized light (LFCP), S-polarized light, P-polarized light, another polarization type, or unpolarized light. The second polarized light may be right circularly polarized (RHCP), left circularly polarized (LFCP), S-polarized, P-polarized, another polarization type, or unpolarized as long as it is different from the first polarized. In some preferred embodiments, the first polarization is one of right circular polarization (RHCP) and left circular polarization (LFCP) and the second polarization is left circular polarization (LFCP) and right circular. It is the other of the polarized light (RHCP).

  In some embodiments, the operation of variable optics 50004 may be electronically programmed to be synchronized with the frame rate of the high FOV low resolution image stream and the low FOV high resolution image stream in time division multiplexing. Good. In some embodiments, the high FOV stream image frame is given its desired polarization and angular magnification and couples to the waveguide 50010 via the ICG50006, while the low FOV stream interleaved frame is Given its desired magnification and polarization, it is first passed through ICG 50006, passed through mirror 50008, targeted to the user's fixation point, and then coupled through ICG 50006 to waveguide 50010.

  The display system 50000 also includes an eyepiece 50010 and a polarization sensitive internal coupling grating (ICG) 50006 coupled to the eyepiece 50010. The eyepiece 50010 may be a waveguide, eg, plate, configured to propagate light therein, eg, by total internal reflection. The polarization sensitive ICG 50006 may be a polarization sensitive diffractive optical element configured to diffract a portion of the light incident upon it into the eyepiece 50010. In some embodiments, the ICG50006 is configured such that incident light having a particular polarization is preferentially diffracted into the eyepiece 50010, while at least one other polarized incident light passes through the ICG50006. In that respect, it may be polarization sensitive. Light that does not couple into the eyepiece 50010 and passes through the ICG 50006 may be directed towards a mirror 50008, which may be a MEMS mirror and configured to switch the polarization of the incident light. As a first example, a polarization sensitive ICG 50006 couples light with right circular polarization (RHCP) into a waveguide while passing light with left circular polarization (LHCP) towards mirror 500008. Can be done. As a second example, a polarization sensitive ICG 50006 may couple light with LHCP into a waveguide while allowing light with RHCP to pass towards mirror 500008.

  In at least some embodiments, the light reflected from mirror 50008 may be directed towards ICG50006. Additionally, reflection of light from mirror 500008 may modify the polarization of the light such that the reflected light has a desired polarization that will be diffracted by ICG50006 and coupled into eyepiece 50010. (For example, the polarization of light is reversed from RHCP to LHCP and vice versa). As an example, if ICG 50006 is configured to couple light with RHCP into eyepiece 50010, the light associated with the high FOV stream is given RHCP by variable optics 50004, and then It may be coupled into the eyepiece 50010. In such an embodiment, the light associated with the low FOV stream may pass through the ICG50006 without the LHCP light then coupling outcoupling into the eyepiece 50001, and instead toward mirror 500008. LHCP may be provided by the variable optics 50004 so that it can be directed. Reflection of LHCP light from mirror 50008 can reverse the polarization of the light to RHCP. The now RHCP light may then strike the ICG 500062 and be coupled into the eyepiece 50010 by the ICG 50006. A similar embodiment applies when ICG50006 is configured to couple LHCP into eyepiece 50010.

  As disclosed herein, mirror 500008 may be a moveable mirror, eg, a scanning mirror, and may act as a foveal tracker. Mirror 50008 may also be controlled and moved / tilted based on the determined fixation position of the user's eye, as discussed herein. The tilting of the mirror 50008 may cause the reflected light to be internally coupled into the waveguide 500010 at different locations, thereby also coupling the light out at different locations corresponding to the location of the fovea of the user's eye. .

  With continued reference to FIG. 50, the light source 50002 may produce a high FOV low resolution (HFLR) image stream and a low FOV high resolution (LFHR) image stream in a time multiplexed manner. In addition, the variable optics 50004 modifies the HFLR image stream such that the HFLR image stream is coupled into the waveguide 50010 by the polarization-sensitive ICG 50006 to provide a specific polarization (equal RHCP) (and Associated angular magnification). Variable optics may modify the LFHR image stream to have different polarizations (equal LHCP) and associated angular magnification. As a result, the LFHR image stream passes through the polarization sensitive ICG50006, reflects from mirror 500008 (inverts the polarization to RHCP and targets the LFHR image to the user's fixation position) and then guided by ICG50006. Coupled into wave tube 50010.

  Optionally, at least one device for switching the polarization state of light may be inserted in the optical path between image source 50002 and ICG50006.

  FIG. 51 illustrates an example implementation of the variable optics 50004. As shown in FIG. 51, the variable optical system 50004 includes a polarizer 50012, a switchable quarter-wave plate (QWP) 50013, a lens 50014, a diffractive wave plate lens 50015, and a diffractive wave plate lens 50016. , Lens 500017. This is simply one possible implementation of variable optics 50004.

  Polarizer 50012 converts the high FOV low resolution image stream and low FOV high resolution image stream from light source 50002 from unpolarized states to polarized states such as S-polarized and P-polarized or RHCP and LHCP polarized states. May be configured to do so.

  The switchable QWP 50013 may be configured to convert the light polarized from the polarizer 50012 into either (1) right circularly polarized light (RHCP) or (2) left circularly polarized light (LHCP).

  After exiting from QWP50013, the light may be incident on lens 50014 and diffractive waveplate lens 50015. The diffractive waveplate lens 50015 may be a geometric phase lens that includes a pattern-matched liquid crystal material. The diffractive waveplate lens 50015 may have a positive refractive power for circularly polarized light having a handedness (RH or LH) that matches its handedness (eg, is a positive lens), and antipodality. May have a negative refractive power for circularly polarized light (eg, a negative lens). The diffractive waveplate lens 50015 may also have the property of reversing the handedness of circularly polarized light. Therefore, when the diffractive wave plate lens 50015 is clockwise and receives the RHCP light from the lens 500014, the diffractive wave plate lens 50015 acts as a positive lens and the light passes through the diffractive wave plate lens 50015. , It will be counterclockwise.

  After exiting the diffractive wave plate lens 50015, the light will be incident on the diffractive wave plate lens 50016 and then on the lens 50017. Diffractive wave plate lens 50016 may operate in a manner similar to that of diffractive wave plate lens 50015. In addition, the handedness of diffractive wave plate lens 50016 may match that of diffractive wave plate lens 50015, in at least some embodiments. By using such an array, the refractive power of the diffractive waveplate lens 50016 will be opposite to that of the diffractive waveplate lens 50015. Therefore, the switchable QWP 50013 provides light with a polarization that matches the diffractive waveplate lens 50015. In one embodiment, the lens 50015 has a positive refractive power and also reverses the chirality of the light. Let's do it. Then, when the subsequent diffractive waveplate lens 50016 receives light, the lens 50015 will have a negative refractive power as it receives light after its handedness is reversed.

  Using an arrangement of the type shown in FIG. 51, tunable optics 50004 provides a first angular magnification factor when switchable QWP 50013 provides light that matches the palm of diffractive waveplate lens 50015. Alternatively (eg, lens 50015 provides positive power while lens 50016 provides negative power), switchable QWP50013 provides light of opposite handedness when the switchable QWP50013 provides light of the opposite hand. Angular magnification may be provided (eg, lens 50015 provides negative refractive power while lens 50016 provides positive refractive power). In other embodiments, the handedness of the two diffractive waveplate lenses 50015 and 50016 may be different.

  52A-52B, additional details regarding example ICG configurations are provided. For example, it should be appreciated that a polarization sensitive ICG may preferentially direct light in a particular lateral direction depending on which side of the ICG the light is incident on. For example, referring to FIG. 52A, light incident on ICG 50006 from below is redirected to the left of the page. However, light incident on the ICG50006 from above would be undesirably directed towards the right of the page, away from the area of the waveguide, from which the light would be outcoupled to the viewer. .. In some embodiments, different ICGs may be used for light incidence from different directions or from the side of the waveguide 50010 to incoupling the light so that it propagates in the desired direction.

  For example, in some embodiments, a display system uses a waveguide with a high FOV low resolution image stream and a low FOV high resolution image stream using a pair of polarization sensitive inductively coupled gratings (ICG) 50006 and 50040. 50010 (which may be an eyepiece). Such an arrangement is, for example, such that light that strikes the ICG from below (in the perspective of FIGS. 50-53B) is coupled into the waveguide 50010 in the desired lateral direction (left), while above. It may be beneficial if light that strikes the ICG from is coupled into the waveguide 50010 in the opposite direction (right). Further details on In-Coupled Gratings (ICGs) can be found in US patent application Ser. No. 15 / 902,927, the contents of which are expressly and completely in their entirety as if set forth by reference in their entirety. Incorporated herein by reference).

  52A-52B schematically illustrate a display system 52000 for projecting an image stream onto a user's eye, which may include two ICGs 50006 and 50040, according to some embodiments of the present invention. In some embodiments, both ICG50006 and 50040 may be configured to couple light of the same polarization type into waveguide 50010. As an example, each of ICG 50006 and 50040 may couple light having left circular polarization (LHCP) into waveguide 50010 while allowing light having right circular polarization (RHCP) to pass. Alternatively, the polarizations may be interchanged.

  As shown in FIG. 52A, optical elements such as those shown in FIGS. 50-51 may provide a high FOV low resolution image stream 50030 with left-handed circular polarization (LHCP). Light 50030 may be incident on ICG 50006. The light 50030 is LHCP and the ICG 50006 is configured to couple the LHCP light into the waveguide 50010 so that the light is coupled into the waveguide 50010 by the ICG 50006.

  As shown in FIG. 52B, optical elements such as those shown in FIGS. 50-51 may provide a low FOV high resolution image stream 50020 with right-handed circular polarization (RHCP) (time multiplexed). 52A, may be interleaved with the image stream of FIG. 52A). Light 50020 may be incident on ICG50006. However, light 50020 is RHCP and ICG 50006 is configured to couple only LHCP light into waveguide 50010, so light 50020 passes through ICG 50006. ICG50040 may also be configured to couple only LHCP light into waveguide 50010, and thus light may also pass through ICG50040. After passing through both ICGs, light 50020 may also be in a particular orientation based on the user's fixation point (as discussed herein in various sections), and may also be incident on movable mirror 500008. Good. After reflection from mirror 500008, the polarization of light 50020 may be inverted, so the light now becomes LHCP. Light 50020 may then be incident on ICG 50040, which may currently couple LHCP light 50020 into waveguide 50010.

  In some embodiments, the display system may be configured such that the high FOV low resolution image stream and the low FOV high resolution image stream are formed by light having the same polarization. As a result, both image streams may be internally combined by the same ICG in response to being incident on the same side of the ICG.

  53A-53B schematically illustrate a display system 53000 for projecting an image stream to a user's eye, which includes a single ICG 50006 and a switchable reflector 50042, according to some embodiments of the present invention. And may be included. The switchable reflector 50042 may be a liquid crystal based planar device that switches between a substantially transmissive state and a substantially reflective state at a sufficiently high rate. That is, the switch rate of switchable reflector 50042 is preferably high enough to allow coordination with interleaved frames of high FOV low resolution image streams and low FOV high resolution image streams. For example, switchable reflector 50042 is preferably capable of switching between a reflective state and a transmissive state at least at the same rate as the high and low FOV resolution image streams are switched.

  As shown in FIG. 53A, the ICG50006 may receive the high FOV low resolution image stream 50030 from optical elements such as those shown in FIGS. 50-51. As an example, the image stream may have left-handed circular polarization (LHCP). The light of image stream 50030 may be incident on ICG 50006. However, the ICG50006 may be configured to combine RHCP light and pass LHCP light. Therefore, LHCP light 50030 may pass through ICG 50006. The light may then be incident on the switchable reflector 50042 (while the system is projecting the high FOV low resolution image stream 50030), which may be configured into its reflective state. Accordingly, the light of image stream 50030 may be reflected from switchable reflector 50042, thereby reversing the handedness of its polarization. After reflection from the switchable reflectors 50042, 50030, the light may again be incident on the ICG50006, which may now couple the RHCP light 50030 into the waveguide 50010.

  As shown in FIG. 53B, optical elements such as those shown in FIGS. 50-51 may provide a low FOV high resolution image stream 50020 with left-handed circular polarization (LHCP). This arrangement differs slightly in that the polarization of the low FOV image stream 50020 matches the polarization of the high FOV image stream 50030. Such an arrangement may be achieved using a modification of the variable optics 50004 shown in Figures 50-51. By way of example, an additional polariser, eg a switchable polariser, may be provided between the lens 50017 and the ICG 50006.

  Returning to the low FOV high resolution LHCP light 50020 in FIG. 53B, the light 50020 is incident on the ICG 50006. However, the ICG50006 is configured to couple the RHCP into the waveguide 50010. Therefore, light 50020 passes through ICG50006. Light 50020 may then be configured to be in its transmissive state (while the system projects low FOV high resolution light 50020) and is incident on switchable reflector 50042. Thus, light may pass through the switchable reflector 50042, impinge on the mirror 50008, and optionally be targeted by the mirror 50008 onto the user's fixation point (discussed herein in various sections. like). After reflection from mirror 500008, the polarization of light 50020 may be inverted, so the light now becomes RHCP. The light 50020 may then be incident on the ICG 50006, which may currently couple the RHCP light 50020 into the waveguide 50010. Mirror 50008 may be configured to provide foveal tracking, and / or allow for different focal lengths of wearable optics 50004 (FIGS. 50-51) to provide a well-focused image that is well spaced from ICG50006. It should be appreciated that

  The various aspects, embodiments, implementations or features of the described embodiments can be used separately or in any combination. Various aspects of the described embodiments can be implemented in software, hardware, or a combination of hardware and software. The described embodiments can also be embodied as computer readable code on a computer readable medium for controlling manufacturing operations or as computer readable code on a computer readable medium for controlling a manufacturing line. .. A computer-readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include read only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. Computer readable media can also be distributed via network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

  The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that specific details are not required to practice the described embodiments. Accordingly, the foregoing description of specific embodiments is presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those skilled in the art that many modifications and variations are possible in light of the above teaching.

  Also, each of the processes, methods, and algorithms described herein and / or depicted in the figures each comprise one or more physical computing devices configured to execute specific and specific computer instructions. It is to be understood that it may be embodied in a code module executed by an embedded system, a hardware computer processor, application specific circuitry, and / or electronic hardware, and thereby fully or partially automated. For example, a computing system may include a general purpose computer (eg, a server) or special purpose computer, special purpose circuit, etc. programmed with specific computer instructions. Code modules can be written in a programming language that can be compiled and linked into an executable program, installed in a dynamically linked library, or interpreted. In some implementations, particular acts and methods may be implemented by circuitry that is specific to a given function.

  In addition, the functional implementations of the present disclosure are sufficiently mathematical, computer, or technically complex that application-specific hardware (using appropriate specialized executable instructions) or One or more physical computing devices may need to implement functionality, for example, due to the amount or complexity of the calculations involved or to provide results in substantially real time. For example, a video may include many frames, each frame having millions of pixels, and specifically programmed computer hardware to achieve the desired image processing task or The video data needs to be processed to provide the application.

  The code module or any type of data may be a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage, combinations of the same, and / or It may be stored on any type of non-transitory computer readable medium, such as physical computer storage, including equivalents. In some embodiments, the non-transitory computer readable medium is part of one or more of a local processing and data module (140), a remote processing module (150), and a remote data repository (160). It may be. The method and module (or data) may also be produced as a data signal (eg, a carrier wave or other analog or digital propagated signal) generated on a variety of computer readable transmission media, including wireless and wire / cable based media. (As part) and may take various forms (eg, as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be persistently or otherwise stored in non-transitory tangible computer storage of any type, or may be communicated via a computer-readable transmission medium.

  Any process, block, state, step or functionality in the flow diagrams described herein and / or depicted in the accompanying figures refers to a specific function (eg, logic or arithmetic) in the process or It is to be understood as potentially representing a code module, segment, or portion of code that contains one or more executable instructions for implementing steps. The various processes, blocks, states, steps, or functionalities may be combined, rearranged, added, deleted, modified, or otherwise illustrative examples provided herein. May be changed from In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith may be in any other suitable sequence, for example consecutively or in parallel. , Or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described in this embodiment is for the purpose of illustration and should not be understood as requiring such separation in all embodiments. It is to be appreciated that the program components, methods, and systems described can generally be integrated together in a single computer product or packaged in multiple computer products.

  In the preceding specification, the present invention has been described with reference to specific embodiments thereof. It will be apparent, however, that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

  Indeed, the systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for, or required for, the desirable attributes disclosed herein. I want you to understand. The various features and processes described above can be used independently of each other or can be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure.

  Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features are described above as operating in some combinations and may be initially claimed as such, one or more features from the claimed combination may, in some cases, be a combination. The claimed combinations may be directed to sub-combinations or variations of sub-combinations. No single feature or group of features is necessary or required for every embodiment.

  In particular, "can", "could", "might", "may", "e.g." and the like. Etc., as used herein, unless stated otherwise specifically, or otherwise understood within the context in which they are used, generally refer to certain features, elements, It should be understood that while including and / or steps, other embodiments are intended to convey that they do not. Accordingly, such conditional statements generally require that features, elements, and / or steps are required for one or more embodiments, or for one or more embodiments. Necessarily involves logic to determine whether these features, elements, and / or steps are included or should be performed in any particular embodiment, with or without author input or prompting. It is not intended to suggest inclusion in. The terms "comprising," "including," "having," and equivalents are synonymous and are used in a non-limiting manner, and include additional elements, features, acts, acts, etc. Do not exclude. Also, the term "or" is used in its inclusive sense (and not in its exclusive meaning), and thus, for example, when used to connect a list of elements, the term "or" Means one, some, or all of the elements in the list. In addition, the articles "a", "an", and "the" as used in this application and the appended claims, unless otherwise specified, are "one or more" or "at least one." Should be interpreted to mean. Similarly, although acts may be depicted in the figures in a particular order, this may be performed in the particular order in which such acts are shown or in a sequential order to achieve the desired result, or It should be appreciated that not all illustrated acts need be performed. Moreover, the drawings may diagrammatically depict one or more exemplary processes in the form of flowcharts. However, other acts not depicted may also be incorporated within the illustrative methods and processes illustrated graphically. For example, one or more additional acts may be performed before, after, concurrently with, or during any of the depicted acts. Additionally, operations may be reordered or reordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems generally It should be appreciated that they may be integrated together in a single software product or packaged in multiple software products. In addition, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

  Therefore, the claims are not intended to be limited to the implementations shown herein and should be given the broadest scope consistent with the disclosure, principles, and novel features disclosed herein. is there.

Claims (128)

A system, wherein the system comprises one or more processors and one or more computer storage media, the one or more computer storage media storing instructions, the instructions comprising: When executed by one or more processors, the one or more processors
Monitoring user eye movements based on information detected via one or more sensors;
Determining a fixation point on which the user's eye is gazing based on the eye movement, wherein the fixation point is a three-dimensional place in the user's visual field;
Obtaining location information associated with one or more virtual objects for presentation to a user, the location information indicating a three-dimensional position of the virtual object;
Adjusting the resolution of at least one virtual object based at least in part on the proximity of the at least one virtual object and the fixation point;
Providing a presentation of the one or more virtual objects to a user via a display device, the at least one virtual object being rendered according to the individually adjusted resolution. A system that implements   The system of claim 1, wherein adjusting the resolution of the virtual object comprises reducing the presentation quality of the virtual object.   Reducing the presentation quality of the virtual object includes reducing the polygon count of the virtual object, adjusting the primitives used to generate the virtual object, and performing the actions performed on the virtual object. 3. The method of claim 2, comprising one or more of adjusting, adjusting texture information, adjusting color resolution, or depth, and adjusting the number of rendering cycles or frame rate. system.   The system of claim 1, wherein the field of view corresponds to a display frustum of the display device. The field of view comprises a plurality of zones, each zone representing a volume of space within the field of view in which a virtual object may be presented by the display device,
The zone extends from an initial depth in the field of view to an end depth in the field of view,
The proximity of the one or more virtual objects from the fixation point corresponds to a distance separating a first zone in which the virtual object is to be presented from a second zone in which the fixation point is located,
The system of claim 1.   Adjusting the resolution of the at least one virtual object comprises reducing the resolution from a resolution initially assigned to the at least one virtual object, the proximity measuring the first zone to the second zone. The system of claim 5, corresponding to the total number of zones that separate from the zones.   The second zone in which the fixation point is located is assigned a maximum resolution, and the remaining zones are each assigned a resolution reduced from the maximum resolution according to one or more descent rates. System of. The operation further includes
Providing a user retinal cone density profile,
6. The system of claim 5, wherein a rate of descent on an individual depth plane substantially conforms to a rate of descent of cone density within the user retinal cone density profile.   The system of claim 5, wherein the zone has a three-dimensional polygonal shape.   The system of claim 5, wherein the zones are concentric ellipsoids.   The system of claim 5, wherein the shape of the zone varies with the distance to the second zone. Further comprising the display device, wherein the display device is
A plurality of stacked waveguides forming a display area and providing a view of the ambient environment through the display area, at least some of the waveguides of the plurality of waveguides being other waveguides. The system of claim 1, comprising a plurality of stacked waveguides configured to output light with different wavefront divergence. With respect to the first virtual object, which is to be presented at a depth farther from the fixation point and is determined to be within a threshold angular distance of the user's line of sight, the operation further comprises:
Adjusting the resolution of the first virtual object based on the degree of proximity between the first virtual object and the fixation point;
Blurring the first virtual object during presentation of the first virtual object to a user;
The system of claim 1, comprising: Blurring the first virtual object includes:
Performing a Gaussian blur kernel convolution, wherein the first virtual object is rendered according to the adjusted resolution;
Presenting the result of the convolution to the user via the display device;
14. The system of claim 13, comprising:   The one or more sensors comprise one or more of an infrared sensor, an ultraviolet sensor, and a visible wavelength light imaging device configured to detect a line-of-sight direction of the user's eye. The system according to 1. A display system, wherein the display system comprises:
A display device configured to present virtual content to a user,
One or more processors,
One or more computer storage media, the one or more computer storage media storing instructions, wherein the instructions when executed by the system include:
Monitoring information associated with the user's eye movements;
Determining a fixation point within a display frustum of the display device based on the monitored information, the fixation point indicating a three-dimensional location being fixed by the eye of the user. When,
Presenting virtual content to a three-dimensional location within the display frustum based on the determined fixation point, wherein the virtual content is based on proximity of the virtual content from the fixation point, That the resolution is adjusted,
One or more computer storage media causing the operation to be performed. The operation further includes
Providing the determined fixation point to an external system comprising one or more processors by the display device;
Receiving, by the display device, from the external system rendered virtual content for presentation, the virtual content at a resolution based on the individual proximity of the virtual content from the fixation point. To be rendered,
17. The system of claim 16, comprising: Adjusting the resolution of a particular virtual content based on the proximity from the fixation point includes:
Determining a reduction in maximum resolution based on the proximity, the reduction being based on a rate of descent;
Adjusting the resolution of the particular virtual content based on the determined reduction.   17. The system of claim 16, wherein the display device comprises a plurality of waveguides, each waveguide presenting virtual content associated with an individual depth based on a three-dimensional location of the virtual content. ..   The display device presents virtual content based on foveated imaging for the fixation point, the foveated imaging being (1) depth information associated with the fixation point and (2) virtual content. 17. The system of claim 16 incorporating the. Method,
With a system of one or more processors,
Monitoring the eye orientation of the user of the display device based on information detected via the one or more sensors;
Determining a fixation point that the user's eye is gazing based on the eye orientation, the fixation point representing a three-dimensional location relative to the user's field of view;
Obtaining location information associated with one or more virtual objects for presentation to a user, the location information indicating a three-dimensional position of the virtual object;
Adjusting the resolution of the at least one virtual object based at least in part on the individual proximity from the at least one virtual object to the fixation point;
Providing a presentation of the one or more virtual objects to a user via the display device, wherein the at least one virtual object is rendered according to the individual adjusted resolutions. ,Method.   22. The method of claim 21, wherein adjusting the resolution of the virtual object comprises reducing the presentation quality of the virtual object relative to a maximum possible presentation quality.   Reducing the presentation quality of the virtual object reducing the polygon count of the virtual object, reducing the primitives associated with the virtual object, or reducing the texture information associated with the virtual object. 23. The method of claim 22, comprising:   22. The method of claim 21, wherein the fixation point is determined for each frame presented to a user via the display device. The field of view comprises zones, each zone representing a volume of space within the field of view in which a virtual object may be presented,
The zone extends at least from an initial depth within the field of view to an end depth within the field of view;
22. The proximity of a virtual object from the fixation point is determined based on a first zone in which the virtual object will be presented to a second zone in which the fixation point is located. Method.   22. The method of claim 21, wherein the sensor comprises one or more of an infrared sensor, an ultraviolet sensor, and a visible wavelength light imaging device. A display system,
A frame configured to be mounted on the user's head,
A light modulation system configured to output light and form an image,
One or more waveguides mounted to the frame and configured to receive light from the light modulation system and output the light across the surface of the one or more waveguides;
One or more processors,
One or more computer storage media, wherein the one or more computer storage media store instructions, the instructions being executed by the one or more processors. To
Determining the amount of light reaching the retina of the user's eye;
Adjusting the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina, and one or more computer storage media.   28. The method further comprising an outwardly directed camera configured to measure ambient lighting levels, and determining the amount of light reaching the retina comprises measuring the ambient lighting levels. Display system according to.   29. The display system of claim 28, wherein determining the amount of light reaching the retina further comprises determining the amount of light output by the waveguide to the user's eye. The operation further includes
Determining the user fixation point,
Acquiring the location information of the virtual content, wherein the location information indicates a three-dimensional position of the virtual content, and adjusting the resolution of the virtual content includes: 28. The display system of claim 27, comprising varying resolution of the virtual content based on proximity of fixation points.   Adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on the proximity of the fixation point and the virtual content on a depth axis extending away from the user, The display system according to claim 30.   The one or more waveguides comprises a stack of waveguides, at least some of the waveguides of the stack of waveguides are different from other waveguides of the stack of waveguides. 28. The display system of claim 27, which provides a wavefront divergence amount. The one or more waveguides are
An internally coupled diffractive optical element,
An externally coupled diffractive optical element,
30. A light dispersive element configured to direct light from the incoupling diffractive optical element to the outcoupling diffractive optical element. A display system,
One or more processors,
One or more computer storage media, wherein the one or more computer storage media store instructions, the instructions being executed by the one or more processors. To
Determining the amount of light reaching the retina of the user's eye of the display system;
Adjusting the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina, and one or more computer storage media.   35. The display system of claim 34, wherein determining the amount of light that reaches the retina comprises determining ambient lighting level.   36. The display system of claim 35, wherein determining the amount of light reaching the retina comprises determining the amount of light output by the display system to display the virtual content.   35. The display system of claim 34, wherein determining the amount of light that reaches the retina comprises determining a size of a pupil of the user's eye.   The display system of claim 34, wherein adjusting the resolution of the virtual content comprises decreasing the resolution of the virtual content as the distance of the virtual content from the user fixation point increases.   39. The display system of claim 38, wherein the tendency to reduce resolution is substantially similar to the tendency to reduce cone density in the user's retina. The zone of maximum resolution within the user's field of view is centered on a point corresponding to the fossa of the user's eye,
The resolution of the virtual content in the user's field of view is reduced for virtual content located +/− 20 ° outside the point,
The display system according to claim 39.   39. The display system of claim 38, wherein reducing the resolution comprises reducing polygon counts of all levels of light reaching the retina.   39. The display system of claim 38, wherein adjusting the resolution comprises setting the resolution to one of two resolution levels.   43. The display system of claim 42, wherein adjusting the resolution of the virtual content comprises reducing one or both of color depth and contrast ratio with decreasing amount of light reaching the retina. The operation further includes
Determining whether the amount of light corresponds to a photopic, mesopic, or scotopic illumination level;
35. Adjusting the resolution of the virtual content comprises setting the resolution based on whether the lighting level corresponds to a photopic, mesopic, or scotopic lighting level. Display system according to.   35. The display system of claim 34, wherein adjusting the resolution of the virtual content comprises displaying the virtual content using monochrome in a scotopic lighting level.   35. A display as claimed in claim 34, wherein adjusting the resolution of the virtual content comprises decreasing the contrast ratio of the images forming the virtual content as the amount of light reaching the retina of the eye decreases. system. The operation further includes
Providing virtual content in multiple primary colors,
35. The display system of claim 34, wherein adjusting the resolution of the virtual content comprises providing different resolutions for different primary colors.   35. The display system of claim 34, wherein the act of further comprises preventing rendering of virtual content corresponding to an optic nerve blind spot in the eye.   Adjusting the resolution of the virtual content includes preventing rendering of virtual content located in opposite peripheral regions for each of the user's eyes, the opposite peripheral region of the user being positioned with the eyes. 35. The display system of claim 34, which is on the side of the user opposite the side.   35. The display system of claim 34, wherein the act further comprises providing a presentation of the virtual content to a user via a display device, the virtual content being rendered according to an associated adjusted resolution. .. A method implemented by a display system comprising one or more processors and a head mountable display, the method comprising:
Determining the amount of light reaching the retina of the user's eye of the display system;
Adjusting the resolution of the virtual content to be presented to the user based on the amount of light reaching the retina. A display system,
A frame configured to be mounted on the user's head,
A light modulation system configured to output light and form an image,
One or more waveguides mounted to the frame and configured to receive light from the light modulation system and output the light across the surface of the one or more waveguides;
One or more processors,
One or more computer storage media, wherein the one or more computer storage media store instructions, the instructions being executed by the one or more processors. To
Proximity of the virtual content from the user's perspective,
The resolution of the primary color image forming the virtual content based on the color of the primary color image, wherein at least one of the primary color images has a different resolution from the primary color image of another color. One or more computer storage media that perform operations including adjusting the display system.   Adjusting the resolution of the virtual content includes varying the resolution of the virtual content based on the proximity of the fixation point and the virtual content on a depth axis extending away from the user, 53. The display system of claim 52.   The one or more waveguides comprises a stack of waveguides, at least some of the waveguides of the stack of waveguides are different from other waveguides of the stack of waveguides. 53. The display system of claim 52, which provides a wavefront divergence amount. The one or more waveguides are
An internally coupled diffractive optical element,
An externally coupled diffractive optical element,
A light dispersive element configured to direct light from the inner coupled diffractive optical element to the outer coupled diffractive optical element;
53. The display system of claim 52, comprising. A display system,
One or more processors,
One or more computer storage media, wherein the one or more computer storage media store instructions, the instructions being executed by the one or more processors. To
Proximity of the virtual content from the user's perspective,
The resolution of the primary color image forming the virtual content based on the color of the primary color image, wherein at least one of the primary color images has a different resolution from the primary color image of another color. One or more computer storage media that perform operations including adjusting the display system.   57. The display system of claim 56, wherein the resolution corresponds to at least color depth.   57. The display system of claim 56, wherein the resolution corresponds to at least polygon count. The primary color image comprises a red primary color image, a green primary color image, and a blue primary color image,
The green primary color image has a higher resolution than the red or blue primary color image,
A display system according to claim 56.   60. The display system of claim 59, wherein the red primary color image has a higher resolution than the blue primary color image.   Adjusting the resolution of the virtual content comprises reducing the contrast ratio of the images forming the virtual content with a decrease in one or both of ambient light and light output by the display system to display the virtual content. 57. The display system of claim 56, comprising:   57. The display system of claim 56, wherein the act further comprises preventing rendering of virtual content corresponding to an optic nerve blind spot in an eye of a user of the display system.   Adjusting the resolution of the virtual content includes preventing rendering of virtual content located in opposite peripheral regions for each of the user's eyes, the opposite peripheral region of the user being positioned with the eyes. 57. The display system of claim 56, which is on the side of the user opposite the side.   57. The display system of claim 56, wherein the act of further comprises providing a presentation of the virtual content to a user via a display device, the virtual content being rendered according to an associated adjusted resolution. .. A method implemented by a display system comprising one or more processors and a head mountable display, the method comprising:
Proximity of the virtual content from the user's perspective,
The resolution of the primary color image forming the virtual content based on the color of the primary color image, wherein at least one of the primary color images has a different resolution from the primary color image of another color. Adjusting the method. A display system,
An image source comprising a spatial light modulator for providing a first image stream and a second image stream;
A viewing assembly including light guiding optics for receiving the first and second image streams from the image source and outputting the first and second image streams to a user.
One or more processors in communication with the image source;
One or more computer storage media, wherein the one or more computer storage media store instructions, the instructions being executed by the one or more processors. To
Causing the image source to output the first image stream to the viewing assembly, the image formed by the first image stream having a first pixel density;
Causing the image source to output the second image stream to the viewing assembly, wherein the image formed by the second image stream has a second pixel density greater than the first pixel density. Have an action including
The image formed by the second image stream corresponds to a portion of the image provided by the first image stream,
The image formed by the second image stream comprises one or more computer storage media covering a corresponding portion of the field of view provided by the first image stream.   67. The display system of claim 66, wherein the act further comprises modifying a location of the second image with respect to the first image.   68. The display system of claim 67, wherein the act further comprises modifying the size of the second image over time. Further comprising an eye gaze tracker configured to detect a change in orientation of the user's eye,
The act includes modifying a location of the second image relative to the first image according to a detected change in orientation of the user's eye.
The display system of claim 67. A scan mirror in the path of the second image stream between the image source and the viewing assembly;
70. The display system of claim 69, wherein the act further comprises adjusting an orientation of the scanning mirror to alter an image location of the second image with respect to the first image. The viewing assembly includes an eyepiece, and the eyepiece includes
67. The display system of claim 66, comprising a waveguide with an incoupling grating configured to incouple light into the waveguide.   72. The display system of claim 71, wherein the in-coupling grating is a liquid crystal polarization sensitive in-coupling grating.   The waveguide comprises another incoupling grating configured to incouple light into the waveguide, the other incoupling grating on a surface of the waveguide different from the incoupling grating. 72. The display system of claim 71 disposed along. Both the incoupling grating and the other incoupling grating are in the optical path of the second image stream,
The incoupling grating and the other incoupling grating are transparent to the second image stream having a first polarization,
Further comprising a mirror spaced from and facing the other in-coupling grating, the mirror reflecting the second image stream towards the other in-coupling grating, and also the second image. Configured to change the polarization of the light in the stream to a second polarization, and the other incoupling grating is configured to incouple the light of the second polarization.
A display system according to claim 73.   The waveguide comprises another incoupling grating configured to internally couple light into the waveguide, the other incoupling grating being on the same surface of the incoupling grating and the waveguide. 72. Along a line, the incoupling grids are located in the path of the second image stream, and the other incoupling grids are located in the path of the first image stream. Display system described.   67. The display system of claim 66, further comprising beam splitting optics for splitting light into multiple image streams propagating in different directions. 77. The display system of claim 76, wherein the beam splitting optics is a polarizing beam splitter.
77. The display system of claim 76, wherein the beam splitting optics is a switchable reflector, and the switchable reflector is selectively switchable between a reflective state and a transmissive state. A display system for projecting an image onto a user's eye, said display system comprising:
An image source, the image source comprising a first light beam associated with a first image stream in a first polarization, and a second image stream in a second polarization different from the first polarization. A second image stream configured to project an associated second light beam, the first image stream having a first field of view and the second image stream being narrower than the first field of view. An image source, the first light beam and the second light beam having a field of view of
A polarization beam splitter, wherein the polarization beam splitter is
Receiving the first beam of light and reflecting it toward a viewing assembly along a first optical path to position the first image stream in a fixed position relative to the user's eye;
A polarization beam splitter configured to receive the second light beam and transmit the second light beam along a second optical path;
A scanning mirror disposed along the second optical path and configured to receive the second light beam and reflect it toward the viewing assembly;
An eye gaze tracker configured to detect movement of the user's eyes,
A control circuit in communication with the eye gaze tracker and the scanning mirror, wherein the control circuit performs the scanning so that the position of the second image stream is moved according to the detected movement of the user's eye. A display system configured to position the mirror.   79. The display system of claim 78, wherein the first image stream has a first angular resolution and the second image stream has a second angular resolution that is higher than the first angular resolution.   79. The display system of claim 78, wherein the content of the second image stream comprises a portion of the content of the first image stream.   The content of the second image stream is such that the content of the second image stream corresponds to the portion of the first image stream overlaid by the second image stream. 81. The display system of claim 80, which varies as it moves with respect to the first image stream.   79. The display system of claim 78, wherein the first polarized light and the second polarized light are each linearly polarized light, and the second polarized light is orthogonal to the first polarized light.   79. The display system of claim 78, further comprising a first relay lens assembly configured to provide a first angular magnification factor to the first light beam. A second relay lens assembly configured to provide a second angular magnification to the second light beam, wherein the second angular magnification is different from the first angular magnification. 84. The display system of claim 83, further comprising a second relay lens assembly.   85. The display system of claim 84, wherein the first angular magnification is greater than 1 and the second angular magnification is 1 or less.   The first relay lens assembly comprises a first optical lens positioned between the image source and the polarizing beam splitter, and a second optical lens positioned along the first optical path, 85. The display system of claim 84, wherein the ratio of the focal length of the first optical lens and the focal length of the second optical lens is greater than 1.   The second relay lens assembly comprises the first optical lens and a third optical lens positioned along the second optical path, the focal length of the first optical lens and the first optical lens 87. The display system of claim 86, wherein the ratio of the focal lengths of the 3 optical lenses is 1 or less. The first light beam and the second light beam are time division multiplexed,
The image source comprises a switched polarization rotator, the switched polarization rotator being configured to provide a first light beam at the first polarization and a second light beam at the second polarization. 79. The display system according to claim 78, wherein:   79. The display system of claim 78, wherein the first light beam and the second light beam are polarization division multiplexed. A display system for projecting an image onto a user's eye, said display system comprising:
An image source configured to project a first light beam associated with a first image stream and a second light beam associated with a second image stream, the first image comprising: The stream has a first field of view, the second image stream has a second field of view that is narrower than the first field of view, and the first light beam and the second light beam are multiplexed. Image source,
A scanning mirror configured to receive the first light beam and the second light beam and reflect them towards a viewing assembly to project the first image stream and the second image stream. When,
An eye gaze tracker configured to detect movement of the user's eyes,
A control circuit in communication with the eye gaze tracker and the scanning mirror, wherein the control circuit detects the detected movement of the eye of the user such that the position of the first image stream and the position of the second image stream. A control circuit configured to position the scanning mirror to be moved according to
A switchable optical element disposed in an optical path of the first light beam and the second light beam, wherein the switchable optical element is configured such that the first light beam has a first angular magnification factor. By a second angular magnification factor that is switched to a first state for the first light beam so that the second light beam is less than the first angular magnification factor so as to be angularly magnified. A switchable optical element configured to be switched to a second state for the second light beam so as to be angularly amplified.   91. The display system of claim 90, wherein the first light beam and the second light beam are time division multiplexed and the switchable optical element comprises an electrically switchable liquid crystal lens.   91. The display system of claim 90, wherein the first light beam and the second light beam are polarization division multiplexed and the switchable optical element comprises a multifocal birefringent lens.   93. The display system of claim 92, wherein the multifocal birefringent lens comprises a birefringent polymer.   91. The display system of claim 90, wherein the switchable optical element is located downstream of the scan mirror.   91. The display system of claim 90, wherein the switchable optical element is located on the surface of the scanning mirror.   91. The display system of claim 90, wherein the switchable optical element is located between the image source and the scanning mirror.   91. The display system of claim 90, wherein the first angular magnification is greater than 1 and the second angular magnification is about 1. A wearable display system,
An afocal magnifying lens with circular polarization handedness dependent magnifying power, wherein the afocal magnifying lens is
A first fixed focal length lens element,
A first geometric phase lens exhibiting a positive refractive power for the first handedness of incident circularly polarized light and a negative power for the second handedness of the incident circularly polarized light;
A wearable display system comprising an afocal magnifying lens, comprising: a second geometric phase lens; When transmitting light, the first geometric phase lens reverses the circular polarization chirality,
The second geometric phase lens exhibits a negative refractive power for the first handedness of the circularly polarized light and a positive refractive power for the second handedness of the circularly polarized light,
The wearable display system according to claim 98.   A second fixed focal length lens is further included, wherein the second geometric phase lens exhibits a positive refractive power for the first handedness of the circularly polarized light, and the second handedness of the circularly polarized light. 99. The wearable display system of claim 98, exhibiting negative refractive power for.   99. The wearable display system of claim 98, wherein the first geometric phase lens reverses circular polarization chirality when transmitting light.   Considering the first fixed focal length lens element and the first geometric phase lens as a lens group, for the light having the first handedness, the lens group has a positive focus of the first group. For light having a distance and having the second handedness, the lens group has a positive focal length of the second group and the positive focal length of the first group has a positive focal length of the second group. 99. The wearable display system of claim 98, wherein the positive focal length of the group is exceeded. A polarization rotator optically coupled to the afocal magnifying lens, the polarization rotator receiving polarized light and selectively outputting first linearly polarized light or second linearly polarized light. A polarization rotator,
A first wave plate between the polarization rotator and the afocal magnifying lens, wherein the first wave plate converts the first linearly polarized light into the first handed circularly polarized light. 100. The wearable display system of claim 98, further comprising: a first waveplate configured to convert and convert the second linearly polarized light to the second handed circularly polarized light. A first optical path between the polarization rotator and the first wave plate for light of the first linearly polarized light;
105. The wearable display system of claim 103, comprising: a second optical path between the polarization rotator and the first waveplate for the second linearly polarized light.   105. The wearable display system of claim 104, wherein at least one of the first optical path and the second optical path comprises a steerable mirror.   105. The wearable display system of claim 104, wherein the first optical path and the second optical path each include at least a subset of a plurality of relay lens elements.   104. The wearable display system of claim 103, wherein a continuous pair of relay lens elements in each of the first optical path and the second optical path forms an afocal compound lens.   105. The wearable display system of claim 104, wherein the first optical path intersects a polarization selective reflector and the second optical path intersects the polarization selective reflector.   109. The first optical path and the second optical path each include at least a subset of four relay lens elements, the four relay lens elements positioned about the polarization selective reflector. Wearable display system according to.   The first optical path intersects the polarization-selective reflector in a first direction, and the second optical path intersects the polarization-selective reflector in a second direction. 108. A wearable display system according to 108.   109. The wearable display system of claim 108, wherein at least one of the first optical path and the second optical path includes a second wave plate and a third wave plate. An optical subsystem for a wearable image projector, comprising:
A polarization-selective reflector,
A set of four lens elements centered around the polarization-selective reflector.   113. The optical subsystem of claim 112, wherein each successive pair of the set of four lens elements forms an afocal compound lens along at least one optical path through the subsystem.   113. The optical subsystem of claim 112, further comprising a polarization rotation switch optically coupled to the polarization selective reflector and the set of four lens elements.   115. The optical subsystem of claim 114, further comprising a plurality of wave plates centered about the polarization selective reflector.   116. The optical subsystem of claim 115, wherein each of the plurality of waveplates is positioned between one of the set of four relay lens elements and the polarization selective reflector.   116. The optical subsystem of claim 115, wherein the plurality of wave plates comprises three wave plates.   116. The optical subsystem of claim 115, further comprising a fovea tracking mirror positioned on the opposite side of the polarization selective reflector from the first of the set of four relay lenses.   119. The optical subsystem of claim 118, further comprising an image scanning mirror positioned on the opposite side of the polarization selective reflector from the second of the set of four relay lenses.   120. The optical subsystem of claim 119, further comprising a light beam source optically coupled to the image scanning mirror.   The light beam source includes a dichroic reflector, a first laser diode that emits light of a first color reflected by the dichroic reflector, and a light of a second color transmitted by the dichroic reflector. 121. The optical subsystem of claim 120, comprising an emitting second laser diode. A display system for projecting an image onto a user's eye, said display system comprising:
An eyepiece, wherein the eyepiece is
A waveguide,
An eyepiece, including an incoupling grating optically coupled to the waveguide;
A first image source configured to project a first light beam associated with a first image stream, the first image stream being on a first surface of the incoupling grating. A portion of the first light beam having a first field of incidence is guided by the incoupling grating to position the first image stream in a fixed position relative to the user's eye. A first image source coupled into the wave tube;
A second image source configured to project a second light beam associated with a second image stream, the second image stream having a second field of view narrower than the first field of view. A second image source having
A scanning mirror, wherein the second light beam is such that the second light beam is incident on a second surface of the incoupling grating opposite to the first surface thereof. A scanning mirror configured to receive and reflect a portion of the second light beam into the waveguide by the incoupling grating;
An eye gaze tracker configured to detect movement of the user's eyes,
A control circuit in communication with the eye gaze tracker and the scanning mirror, wherein the control circuit performs the scanning so that the position of the second image stream is moved according to the detected movement of the user's eye. A display system configured to position the mirror.   Further comprising a first lens assembly, wherein the first lens assembly is positioned along a first optical path of the first light beam and provides a first angular magnification to the first light beam. 123. The display system of claim 122, configured to:   A second lens assembly is further provided, the second lens assembly being positioned along a second optical path of the second light beam to provide a second angular magnification to the second light beam. 124. The display system of claim 123, wherein the second angular magnification factor is different from the first angular magnification factor.   125. The display system of claim 124, wherein the first angular magnification factor is greater than 1 and the second angular magnification factor is 1 or less.   123. The display system of claim 122, wherein the first image stream has a first angular resolution and the second image stream has a second angular resolution that is higher than the first angular resolution. A display system for projecting an image onto a user's eye, said display system comprising:
An eyepiece, wherein the eyepiece is
A waveguide,
An eyepiece, including an incoupling grating optically coupled to the waveguide;
An image source, the image source comprising a first light beam associated with a first image stream in a first polarization, and a second image stream in a second polarization different from the first polarization. A second image stream configured to project an associated second light beam, the first image stream having a first field of view and the second image stream being narrower than the first field of view. An image source, the first light beam and the second light beam having a field of view of
A polarization beam splitter, wherein the polarization beam splitter is
Receiving the first light beam and reflecting it along a first optical path;
A polarization beam splitter configured to receive the second light beam and transmit the second light beam along a second optical path;
A first optical reflector, wherein the first optical reflector is positioned along the first optical path and the first light beam is on the first surface of the incoupling grating. Configured to receive and reflect the first light beam for incidence, a portion of the first light beam directing the first image stream to a fixed position relative to the user's eye. A first optical reflector coupled into the waveguide by the incoupling grating for positioning;
A scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam;
A second optical reflector, wherein the second optical reflector is positioned along the second optical path downstream of the scanning mirror, and the second optical reflector is the second optical reflector. Configured to receive and reflect the second light beam such that the light beam is incident on a second surface of the incoupling grating opposite the first surface thereof. A second optical reflector in which a portion of the light beam is coupled into the waveguide by the incoupling grating;
An eye gaze tracker configured to detect movement of the user's eyes,
A control circuit in communication with the eye gaze tracker and the scanning mirror, wherein the control circuit performs the scanning so that the position of the second image stream is moved according to the detected movement of the user's eye. A display system configured to position the mirror. A display system for projecting an image onto a user's eye, said display system comprising:
A waveguide,
An image source, the image source comprising a first light beam associated with a first image stream in a first polarization, and a second image stream in a second polarization different from the first polarization. A second image stream configured to project an associated second light beam, the first image stream having a first field of view and the second image stream being narrower than the first field of view. An image source, the first light beam and the second light beam having a field of view of
A polarization beam splitter, wherein the polarization beam splitter is
Receiving the first light beam and reflecting it along a first optical path;
A polarization beam splitter configured to receive the second light beam and transmit the second light beam along a second optical path;
A first incoupling prism positioned along the first optical path and adjacent a first surface of the waveguide, the first incoupling prism being the first image stream. A first incoupling prism configured to couple a portion of the first light beam into the waveguide for locating a fixed position with respect to the user's eye;
A scanning mirror disposed along the second optical path and configured to receive and reflect the second light beam;
A second incoupling prism positioned adjacent the second surface of the waveguide opposite the first surface of the waveguide along the second optical path downstream of the scanning mirror. Wherein the second incoupling prism is configured to couple a portion of the second light beam into the waveguide;
An eye gaze tracker configured to detect movement of the user's eyes,
A control circuit in communication with the eye gaze tracker and the scanning mirror, wherein the control circuit performs the scanning so that the position of the second image stream is moved according to the detected movement of the user's eye. A display system configured to position the mirror.