WO2014151877A1 - Holographic head-mounted display - Google Patents

Holographic head-mounted display Download PDF

Info

Publication number
WO2014151877A1
WO2014151877A1 PCT/US2014/026603 US2014026603W WO2014151877A1 WO 2014151877 A1 WO2014151877 A1 WO 2014151877A1 US 2014026603 W US2014026603 W US 2014026603W WO 2014151877 A1 WO2014151877 A1 WO 2014151877A1
Authority
WO
WIPO (PCT)
Prior art keywords
slm
image
eye
method
light
Prior art date
Application number
PCT/US2014/026603
Other languages
French (fr)
Inventor
Tomas CIZMAR
Georges A. GOETZ
Daniel V. Palanker
Original Assignee
The Board Of Trustees Of The Leland Stanford Junior University
University Court Of The University Of St. Andrews
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US201361783170P priority Critical
Priority to US61/783,170 priority
Application filed by The Board Of Trustees Of The Leland Stanford Junior University, University Court Of The University Of St. Andrews filed Critical The Board Of Trustees Of The Leland Stanford Junior University
Publication of WO2014151877A1 publication Critical patent/WO2014151877A1/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • G02B27/0172Head mounted characterised by optical features
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03HHOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00Holographic processes or apparatus using light, infra-red or ultra-violet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/22Processes or apparatus for obtaining an optical image from holograms
    • G03H1/2294Addressing the hologram to an active spatial light modulator
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/014Head-up displays characterised by optical features comprising information/image processing systems
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • G02B27/0172Head mounted characterised by optical features
    • G02B2027/0174Head mounted characterised by optical features holographic
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/48Laser speckle optics; Speckle reduction arrangements
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03HHOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00Holographic processes or apparatus using light, infra-red or ultra-violet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/22Processes or apparatus for obtaining an optical image from holograms
    • G03H1/2249Holobject properties
    • G03H2001/2252Location of the holobject
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03HHOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00Holographic processes or apparatus using light, infra-red or ultra-violet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/22Processes or apparatus for obtaining an optical image from holograms
    • G03H1/2249Holobject properties
    • G03H2001/2252Location of the holobject
    • G03H2001/2255Holobject out of Fourier or hologram planes
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03HHOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H2225/00Active addressable light modulator
    • G03H2225/30Modulation
    • G03H2225/32Phase only

Abstract

We provide a head-mounted display unit based on the principle of Fourier holography. It allows for high-efficiency, parallel delivery of information to the retina or to a photo-sensitive implant in the eye. It offers a simpler optical layout than traditional liquid crystal display (LCD) or digital micromirror device (DMD) display units. It addresses the issues associated with an uncontrolled fraction of the light in holography known as the zero order. It provides a framework for displaying visual information on the back of a patient's eye.

Description

Holographic Head-mounted Display

FIELD OF THE INVENTION

This invention relates to projection systems suitable for providing images to the eye of a patient.

BACKGROUND

Current head-mounted displays suffer from a number of shortcomings which limit their usefulness in the context of low- vision restoration. Photovoltaic or optogenetic approaches require irradiance levels on the retina on the order of a few mW/mm2, while traditional head-mounted displays based on LCD or DMD technology provide

illumination on the order of a few nW/mm2, many orders of magnitude below what is required for sight restoration. LCD and DMD technology are also inherently wasteful with the incident light. Additionally, these display systems require the use of multiple optical elements in order to properly focus an image on the back of the eye, which tends to make them bulky. These systems are not inherently able to account for imperfections in the viewer's eye, which means additional eyewear can be required in order to properly focus an image on the retina, for example when the user suffers from near-sightedness or astigmatism.

Furthermore, the image is projected on a fixed plane on the back of the eye, and is inherently flat (two-dimensional) rather than the more natural 3 -dimensional shape of real objects.

Holographic techniques allow for projection of three dimensional patterns, and can account for imperfections in the viewer's eye. They make use of the light in very efficient ways, allowing for high irradiance levels on the target image plane despite limited source power. They also do not require any intermediary optics between the display and the eye, enabling simpler and more compact layouts. However, conventional holographic projection techniques also require a bulky optical system where unwanted diffraction orders are removed by forming an image at an intermediate image plane where an aperture is located. Accordingly, it would be an advance in the art to provide improved projection systems for vision improvement.

SUMMARY We provide a head-mounted display unit based on the principle of Fourier holography. It allows for high-efficiency, parallel delivery of information to the retina or to a photo-sensitive implant in the eye. It offers a simpler optical layout than traditional liquid crystal display (LCD) or digital micromirror device (DMD) display units. It addresses the issues associated with an uncontrolled fraction of the light in holography known as the zero order by the use of converging or diverging illumination of the spatial light modulator. It provides a framework for displaying visual information on the back of a patient's eye.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A shows a conventional 4f Fourier optical system.

FIG. IB show incorporation of the eye into a Fourier optical system.

FIG. 2A shows defocusing of zero order light for a diverging input beam.

FIG. 2B shows defocusing of zero order light for a converging input beam.

FIG. 2C shows simultaneous defocusing of zero order light and focusing of first order light for a diverging input beam.

FIG. 2D shows simultaneous defocusing of zero order light and focusing of first order light for a converging input beam.

FIG. 3A shows an image of defocused zero order light.

Fig. 3B shows a focused image using first order light in the presence of an unfocused zero order background.

FIGs. 4A-B show in vivo experimental results. DETAILED DESCRIPTION

In section A, general principles relating to various embodiments of the invention are described. In section B, an experimental demonstration is described.

A. GENERAL PRINCIPLES

This work provides a head-mounted holographic display unit with potential applications to photovoltaic retinal prosthesis, optogenetic retinal activation, or more generally near-to-eye and augmented reality display systems.

A spatial light modulator (SLM) is placed in front of a person's eye. The eye's natural optics, including the cornea and the lens creates a Fourier transform of the wavefront coming out of the SLM, thus projecting and forming an image on the retina. Polarized light is sent into the display unit using either a fiber which can be polarization- maintaining, or with a light source directly mounted to the side of the display. The light source can either be a coherent, narrow-bandwidth emitter such as a laser or an incoherent source (white light, colored or white LED).

SLMs operate typically in reflection mode. The majority of the light can be controlled by the holographic modulation applied via a computer interface, but there is a fraction of the signal usually termed as "zero diffraction order" or "zero order", that manifest itself as a very bright spot in the image plane (retina) regardless of the holographic modulation applied. This can degrade the imaging performance of the device and can also be potentially hazardous. In the majority of SLM applications it is removed in the optical pathway by blocking it, using for example a slit or an iris. This however requires additional optics and a much longer optical path. To eliminate the influence of the zero diffraction order but retain compactness of the device, our geometry employs a non-collimated laser beam (converging or diverging) to illuminate the SLM, which is further collimated by the SLM using appropriate phase modulation. In this arrangement the optical power of the zero-order spreads across extensive area of the retina thereby avoiding hazardous power concentration, and causing minimal disturbances to the image quality. In the presence of aberrations or misalignment, the resulting field transformation may deviate from an ideal Fourier transform, the holographic modulation can however be adjusted so the system provides an optimal image on the retina. For applications requiring color imaging or specific wavelengths for activation of light-sensitive ion channels, several light sources with different wavelengths can be combined into the same optical fiber or on the same optical path in order to create color images. A

synchronization mechanism is then used for activation of each light source and the SLM, so that the different wavelengths recombine adequately across the image plane.

The light sent into the unit is incident on a beam-splitter, which deflects a fraction or the entire beam onto the SLM. The beam splitter can be a prism, a dichroic mirror or, more generally, any optical element configured to reflect all or a fraction of the light incident on it, either at a specific wavelength, combination of wavelengths or across a broad spectrum. To maximize the efficiency of the system a polarizing beam-splitter may be used in combination with a polarization insensitive SLM and a quarter-wave plate (placed between the beam-splitter and the SLM).

The electronics required to drive the SLM can either be mounted on the goggles or put in an external unit, destined to be worn by the user, for example in a pocket.

A mobile computer/personal phone can be used to compute the holograms displayed on the SLM, or to communicate with a server which does the hologram computation if more computational power is needed, or if remotely stored information has to be included in the hologram, for example for navigation purposes. For retinal prosthetic applications, the phone uses a camera mounted on the glasses to image what is in front of the person. Signal processing software extracts meaningful information from the scene, and computes a relevant hologram that needs to be displayed on the SLM. More generally, for augmented reality applications, signal-processing software computes a hologram that corresponds to the information which needs to be displayed. Images are fed into the SLM at video rates. The hologram computation module could store several pre-computed holograms corresponding to basic information (dots, lines, circles, letters or such) and combine them adequately to reconstruct an image, for example by sequential display of the modules, in order to reduce the computational load. Using a fast SLM, it is possible to display these elements sequentially faster than the perceptual fusion rate of the person, so that they effectively appear as a single image. The camera can be used to extract contextual information relative to positioning of a person, for example, to help patients with low vision navigate around obstacles. The image to be displayed can be broken into subframes which are sequentially activated in order to increase the irradiance locally. The controller used to compute the holograms or the signal processing unit or another dedicated image processing computer can also adjust the power of the light source. This allows maintaining constant brightness levels from frame to frame, or adjusting the scene brightness.

When coherent light illumination is used, speckling can become a serious problem. Several mechanisms to deal with speckling can be used: (a) displaying sequentially different holograms corresponding to the same image realization, (b) changing the spectral characteristics of the light source over time, for example by modulating the wavelength or the emission beam mode. This can be done by ramping up the current which drives the light source, or using a light source specifically designed to have instability in its emission mode or wavelength.

Random light redistribution occurs when pixels are transitioning from one state to another between the frames. Unlike the DMD-based image projection systems, where image from one frame smoothly transitions into the image of the next frame, in SLM projection systems the light spreads randomly during the pixels transition time, resulting in 'flashes' of light in the image background. To overcome this problem, either the SLM is synchronized with the light source(s), so that the light is turned off when images are changing, or the frames are changed frequently enough so that the light redistribution appears as a constant background, virtually invisible to the user. In order to minimize this problem, the algorithm used to compute the holograms can also be tuned to minimize how much sequential holograms change from frame to frame. For example, instead of initializing the GS algorithm with a random-mask encoding of the image, the GS algorithm could be initialized by the previously computed hologram in order to force the next frame to be a hologram which is more similar to the previous one. This initialization method could also offer benefits in terms of the computation speed. The algorithm could also compute several different holograms for each frame and choose the one which minimizes the change compared to the previous hologram.

This approach makes it possible to: 1) Deliver information to multiple retinal cells or pixels of the retinal implant in parallel, with high efficiency. Unlike display systems based on more traditional technologies such as LCD or DMD, no light incident on the display is lost. This visual information can be used to activate a photovoltaic retinal prosthesis, cells made sensitive by optogenetic manipulation, or more generally deliver contextual information to a low- vision patient, or to a normally-sighted person.

2) Provide very broad flexibility in how the light is used. This approach makes it possible to achieve much higher brightness of the image when compared to LCD or DMD displays. By choosing an appropriate hologram, the light incident on the SLM can be concentrated in a small area, even as small as a diffraction limited spot, which makes it possible to reach very high irradiances despite a limited light source power.

3) Not all the light incident on SLM can be effectively controlled by the displayed hologram. The uncontrolled fraction of the light (zero diffraction order) can be distributed as an out-of-focus spot, either in front of or behind the image plane. 4) Have a very simple optical layout. Because only the optics of the eye is used, there is no intermediary lens or other image forming elements other than the beam-splitter and the quarter-wave plate for illumination, which enables a compact display system.

5) Project three dimensional patterns on the retina. It is possible to project the image either slightly in front of the image plane, or behind it, to allow the viewer's eye accommodate and focus on the image, thereby providing a three dimensional perception of the object.

6) Correct for the aberrations of the viewer's eye and systemic aberrations of the optical system. A phase mask compensating for these aberrations can be added to the hologram in order to account for imperfections in the system or in the user's eye (near or far- sightedness, astigmatism, for example).

7) Correct for non-uniform illumination of the SLM when computing holograms, which allows for higher image quality.

8) Reduce the influence of speckling by displaying frames sequentially at a high speed, or by modifying the spectral properties of the light source (wavelength, emission beam mode, etc.). 9) Deal with the random light redistribution which occurs during transition of the pixels between the frames by displaying sequences of holograms fast, or in sync with the pulsed light source. To minimize this effect, the hologram generation algorithm can also be tuned to minimize changes in the SLM state between successive hologram presentations. 10) Reduce computational load when calculating the hologram by recombining several basic elements used to encode a frame, for example lines, dots, circles, letters. These elements can be moved across the field of view by applying an adequate linear modulation and combined by sequential display of the elements. Use of a SLM with a fast refresh rate makes it possible to display the elements faster than the eye will refresh the image, so that the image appears as a single element to the viewer.

The reflective version of the head-mounted unit includes a light source, a beamsplitter which can (but does not need to) be polarizing, a quarter wave plate if the beamsplitter is polarizing, a spatial light modulator which can be polarization-insensitive. With a transmissive SLM, the beam splitter is not required and only a mirror, possibly dichroic or tuned to specific wavelengths, is used to deflect the light on the SLM. The optics are preferably mounted on goggles. Electronics for driving the SLM are also preferably mounted on the goggles or worn in a separate unit. The light source can either be mounted and collimated directly on the side of the beam-splitter, or can be put in a separate unit, and coupled into a fiber. In this case, the fiber output is directed into the beam splitter after partial collimation. The light source can either be a coherent light source (laser-like with a narrow bandwidth), or an incoherent one (white light, LED).

Whether the light sent into the beam splitter comes directly from the light source, or was preliminarily coupled into a fiber, the beam incident on the SLM is purposefully not perfectly collimated. The beam can be either divergent or convergent. The maximum amount of divergence allowed in the beam is related to the resolution of the image, and maximum phase shift on each pixel of the SLM. The beam is collimated after reflecting off (or transmitting through) the SLM by displaying a quadratic modulation on the SLM which acts as a Fresnel lens (see figure 2). Because the pixel pitch of the SLM is finite, large quadratic modulations will exhibit aliasing effects which should be avoided. The largest modulation before such effects appear follows the following equation:

Figure imgf000009_0001
where (|)max is the maximum phase shift of the SLM per pixel, npx is the number of pixels across the largest dimension of the SLM and φ is the quadratic modulation function, so φ η) = an2. This leads to a maximum acceptable value for a of:
Figure imgf000009_0002

and the corresponding focal length for the Fresnel lens is:

2

W

f =—

16D where

Figure imgf000009_0003
λ is the illumination wavelength and w is the width of the SLM across its largest dimension. Consequently, the maximal acceptable beam divergence on the SLM is:
Figure imgf000009_0004

2f w 2jtw

The spot radius on the retina can be approximated by r = θ where Θ is the incident beam divergence angle and/the focal length of the human eye. The minimum

P '

beam divergence is then such that < ImaxW, where Imax(^) is the maximum permissible ocular irradiance for the wavelength chosen and Pi the incident light power on the eye.

Then:

Figure imgf000009_0005
The layout described here offers several significant benefits compared to conventional display systems.

The SLM display uses no intermediary optics between the SLM and the eye, making the layout simpler and potentially more compact, compared to traditional LCD or DMD-based systems which use a set of image-forming lenses between the display and the eye.

Purposefully defocusing the zero order by projecting a non-collimated beam on the SLM helps to safely redistribute the uncontrolled fraction of light incident on the display system (zero order). Using a non-collimated beam incident onto the SLM means that a specular reflection (zero order) from the SLM, including the state when it is turned off, is spread over a large area on the retina, rather than directed into a diffraction limited spot, which would cause safety concerns. A quadratic modulation is used on the SLM to collimate the fraction of the light which can be controlled by the display, thereby allowing image display in the desired plane. The quadratic modulation can also be used to simulate three dimensional images, by creating the image slightly in front of or behind the retina. The modulation acts as a Fresnel lens whose power can be adjusted. When this power changes, the image plane in which the image is created also changes. The user's eye then needs to adjust its focal length to make the image sharp, creating a perception of a 3D vision. The SLM can also be used to correct for systematic aberrations in the optics and imperfections in the user's eye (near, far sightedness, presence of cataracts, etc.).

Another inherent advantage of the system over traditional LCD or DMD based systems is that it uses incident light with much high efficiency since it does not discard any of the optical power, like the LCD or DMD systems, leading to extended battery life. Variations of the given examples include, but are not limited to the following.

The head-mounted unit can be built using either a polarization sensitive or a polarization-insensitive SLM. The polarization-sensitive version of the display requires a non-polarizing beam-splitter and no quarter- wave plate between the beam-splitter and the SLM. The polarization-insensitive version requires a polarizing beam-splitter and the addition of a quarter wave plate between the beam-splitter and the SLM. The light source can either be mounted directly on the display unit (it could for example be a laser diode), or can be kept away from the unit, coupled into a fiber and the fiber delivers the light to the unit. The light can be partially collimated either with a single lens or using a pair of cylindrical lenses, as the distortions introduced by using these can be easily corrected on the SLM itself. If the light source is kept off the display unit, the fiber has to be polarization maintaining, or there has to be a polarizer at the output of the fiber in order to keep control over the polarization of the light inside the display unit. With the second solution, a mobile computing unit predicts the polarization state of the output when the fiber moves, and adjusts power accordingly. The light source can either be coherent, with good spectral purity and single mode emission (single mode laser diode, other type of laser), be coherent with purposefully degraded spectral characteristics in terms of bandwidth (e.g., to reduce speckle), or be incoherent (colored LED, white light).

The mobile processing unit could either be any mobile computing unit running an app dedicated to feeding the right display into the SLM by extracting meaningful information and computing holograms corresponding to it, or could be dedicated electronics optimized for the problem of computing relevant holograms efficiently (ASIC or FPGA circuit implementing a version of the GS algorithm, or another adapted algorithm for hologram computation for example). In the context of a device destined to assist people with low vision, relevant information could be contour extraction, detection and highlighting of an obstacle, extraction of text (single letters could be sharpened, increased in size, text written in small fonts could be broken into several pieces of subtext displayed magnified), signaling of dangers such as cars, assistance with spatial navigation, determination of when it is safe to cross a street and when it is not, assistance in locating specific objects, for example specific food items in a store, or directing someone to their shoes or clothing items. Such embodiments can make use of computer vision methods for recognizing relevant features, which can then have corresponding holograms generated by computation. Such computation can include lookup and/or transformation of pre-computed holograms for commonly encountered relevant features. The mobile processing unit could include a touch-based or voice-based interaction system, which lets the user select what type of visual information is most meaningful to him. The user can change the setting, for example when walking in the street the user could choose to highlight dangers, and when going into a store, could switch to finding a relevant product. The mobile processing unit can be made modular, using different sensors to perform different tasks (camera for detection of obstacles or contour for example, GPS positioning combined with camera for positioning, use of RFID chips for positioning a product in a store).

The signal processing unit can have an internet connection used to communicate with a server in order to improve its recommendations.

In addition to the preceding examples, exemplary embodiments of the invention include the following.

A method for providing an image to an eye of a patient, where the method includes: 1) receiving an input image; 2) computing a hologram corresponding to the input image; 3) encoding the hologram on a phase-only spatial light modulator (SLM); and 4) illuminating the SLM with an optical beam such that phase modulated light from the SLM is received by the eye; where an output image corresponding to the input image is formed in the eye, and where focusing of the optical beam is configured such that zero order light from the SLM is out of focus at a location of the output image.

The eye can include a photoreceptor (e.g., retina or retinal prosthesis) disposed at a photoreceptor location where the output image is to be formed, and the SLM can be disposed at a Fourier conjugate location of the photoreceptor location. Here two locations are Fourier conjugate locations with respect to an optical system if the electric field at one of the locations is the Fourier transform of the electric field at the other location in the optical system. For example, if the optical system is a simple lens with focal length f, planes at distances of +f and -f with respect to the lens are Fourier conjugate locations. In this work, the relevant optical system for defining Fourier conjugate locations is the eye. The SLM can also be disposed away from the Fourier conjugate location of the photoreceptor location. If this is done, appropriate phase modulation applied to the SLM can compensate for this change in SLM position. In this work, a retinal prosthesis is any intra-ocular device capable of receiving light and providing stimulation of optical nerves responsive to the received light. Preferably the optical beam has a focusing angle Θ such that 0min < |θ| < 0max, where 0min and 0max are defined above. Here the beam focusing angle is defined to be the angle of beam convergence or divergence. Either sign convention for 0 for

distinguishing convergence from divergence can be employed without affecting the above result for the preferred focusing angle range, because of the absolute value in the inequality.

The output of the SLM is preferably substantially in the zero order and a single non-zero diffraction order. For example, more than 70% output in the desired diffraction order and less than 10% total output in all diffraction orders other than the desired order and the zero order. Here the zero order has its customary meaning of referring to light that has its propagating wavefront unchanged despite the presence of a phase modulating structure. In this work, the relevant phase modulating structure is the SLM, whose pixels form a periodic array of phase modulating elements. A nematic liquid crystal spatial light modulator configured to control the distribution of incident light among output diffraction orders is a preferred option for the SLM. SLMs with this technology can provide continuous phase shifts (as opposed to binary 0 and 180 degree phase shifts as in ferroelectric SLMs). Using this capability of continuous phase shifts, nematic SLMs can be configured to preferentially provide output in a single non-zero diffraction order (this is analogous to blazing a diffraction grating by adjusting the groove shape of the grating). A quadratic phase shift can be provided with the SLM in order to make the output image three-dimensional.

Speckle removal can be performed. Suitable methods include, but are not limited to: modulating a source of the optical beam, and using multiple holograms for each input image. Method can be employed to mitigate frame switching artifacts. Suitable methods include, but are not limited to: initializing computation of a current frame hologram using a hologram from a previous frame; synchronizing a source of the optical beam and the SLM such that the SLM is not illuminated during frame transitions, and providing a frame rate greater than 20 frames per second (more preferably 40 frames per second or more). B. EXPERIMENTAL DEMONSTRATION Bl. Introduction

Retinal degenerative diseases such as age-related macular degeneration or retinitis pigmentosa are among the leading causes of blindness in the developed world. These diseases lead to a loss of photoreceptors, while the inner retinal neurons survive to a large extent. Activation of the remaining neurons can produce visual percepts, known as phosphenes, thereby enabling delivery of information to the visual system in a blind patient. Electrical stimulation of the surviving retinal neurons has been achieved either epiretinally, in which case the primary targets of stimulation are the retinal ganglion cells (RGCs), or subretinally to bypass the degenerated photoreceptors and use neurons in the inner nuclear layer (bipolar, amacrine and horizontal cells) as primary targets.

Recent clinical studies with epiretinal and subretinal prosthetic systems have demonstrated improvements of the visual function in certain tasks, with some patients being able to identify letters with equivalent visual acuity of up to 20/550. However, both of these systems suffer from a number of shortcomings. Recently approved by the FDA, the ARGUS II implant (Second Sight Inc.) delivers information from an external camera and an external signal processing unit via serial RF telemetry to a sub-conjunctival receiving coil connected to a signal processing module, which then relays the stimulation signals to the epiretinal electrode array via a trans-scleral cable. Such a design is difficult to scale to a much larger number of electrodes than the 60 in the current model. The bulky receiving coil and processing electronics of the implant make the surgery very difficult, and together with the penetrating intraocular cable make it prone to multiple surgical complications. Furthermore, retinal stimulation in this system is directly determined by images captured by the external camera, disregarding the use of natural eye movements to scan a visual scene, a crucial feature of normal visual perception. Implants with photosensitive pixels, such as that from Retina Implant AG, largely overcome the scalability limitation and make use of natural eye movements, but still require external power delivered via RF coils and a trans-scleral cable.

A fully optical design overcomes these limitations by using a subretinal stimulating arrays of photodiodes powered by pulsed near infrared (NIR, 880-915 nm) illumination. This approach enables parallel optical transmission of visual information to each pixel in the implant, adjustable stimulation parameters to modulate retinal response, and preservation of the natural link between eye movements and visual information. Because the photodiode arrays are operated photovoltaically they do not require any wired power connections, greatly simplifying surgery and reducing complications associated with trans-scleral cables. Each array measures 1.2 x 0.8 mm in size and 30 μιη in thickness, and multiple arrays can be tiled in order to increase the field of view. Other fully optical approaches to restoration of sight include optogenetics, in which retinal neurons are transfected to express light-sensitive Na and CI channels, small- molecule photoswitches which bind to K channels and make them light sensitive or photovoltaic implants based on thin-film polymers. Since all these optical approaches require much brighter illumination than the ambient light can provide, a head- mounted display is required to deliver very bright images to the retina. This system should typically include a camera, a signal processing unit and a near-eye display operating at a wavelength optimized for the particular approach. The camera provides autofocusing and adaptation to a broad range of ambient brightnesses, a feature necessary for all stimulation approaches as they have a much narrower dynamic range than the natural variation of lighting conditions. In addition, this approach provides flexibility for image processing between the camera and the display, which is likely to vary for various target cells and techniques of stimulation, and might require optimization for each patient. For activation of the photovoltaic silicon implants NIR light (880-915 nm) with peak irradiance levels of several mW mm is required to elicit retinal response. For other optical approaches very bright UV, blue, green or yellow light is required.

To enhance portability of the system, reduce its weight and maximize battery life, it is important to minimize optical losses and attain the highest efficiency possible. At the same time, the field of view, resolution, and contrast have to be sufficiently high to create meaningful percepts. Finally, ocular safety concerns ought to be kept in mind when designing a system operating with intense illumination, which is often close to the maximum permissible exposure, to ensure that in the event of a critical failure of the display, the user will not be harmed. We describe an optical system based on Fourier holography which provides high power efficiency, very high peak irradiance and allows a sufficiently compact design to be mounted on goggles. We address challenges associated with this geometry, such as the presence of non-controlled light in a zero diffraction order, and complex hologram computation. We show that this system provides good image quality with an acceptable field of view, and demonstrate that image speckling, a very common problem associated with the use of coherent light in imaging, is negligible with our design. We provide a brief safety analysis of the display illustrating the outcomes of potential failure of various components. We provide comparisons with a more conventional LCD- based imaging system. Finally, we demonstrate an application of a slit-lamp mounted version of this holographic projection system through the measurement of cortical response to alternating gratings in normal-sighted rats. Our experiments are tailored to finding an optimal solution for addressing photovoltaic retinal prosthesis, but the principles are applicable for optogenetic or other purely optical sight restoration techniques as well.

B2. System description

B2.1. General principles Common near-the-eye retinal projection systems (video goggles) are based on modulation of brightness in the image, either by attenuating light intensity in each pixel of the liquid crystal displays (LCD) or by varying the duty cycle of each micro-mirror pixel with digital micro-mirror devices (DMD), as is the case in digital light processing (DLP) technology. Both technologies block and discard all undesired optical power, and within the optical system the irradiance modulating element and the retina are placed in conjugate planes.

The system we designed is a modified version of a standard Fourier holographic geometry based on a nematic phase-only spatial light modulator (SLM), which adjusts the phase of the incident wavefront in each pixel. The phase-modulated optical field is transformed into an image by a lens. Conveniently, the transformation between the modulated optical field and the resulting image can be mathematically described by the Fourier transform, which is at the heart of the algorithms for design of holographic modulations. Importantly, instead of blocking light from dark image zones, holographic systems redistribute the optical power across the image with a very high efficiency, thereby greatly reducing power losses in the image formation process, compared to LCD or DMD displays. A typical layout for a Fourier holographic system is shown in FIG. 1A. In such a system, the incident wavefront is modulated by an SLM, and a Fourier lens creates the Fourier transformation of the reflected wavefront in an intermediate image plane. More specifically, a laser beam collimated by a lens (LI) is incident on a (SLM). A Fourier lens (L2) creates an image in an intermediary image plane, where a physical aperture (S) blocks unwanted diffraction orders. A telescope (L3, L4) then projects only the first diffraction order onto the image plane (I). Because of the limited diffraction efficiency of the SLM, undesired diffraction orders, the zeroth order in particular, subsist here. The zero diffraction order affects the central region of the image plane, therefore the image is usually generated off-center in a spatial region associated with the first diffraction order. This is achieved using a blazed grating, analogous to a prism, superimposed onto the hologram phase modulation, which shifts the first diffraction order to the side of the image. Conveniently, all undesired diffraction orders, including the zeroth order, can be removed by spatial filtering using a physical aperture in the image plane, which blocks all except the first order. The final image is projected onto the focal plane using a telescope. Miniaturizing this classical '4f Fourier system into a near-to-eye unit is very challenging from an engineering point of view.

To reduce the size of the holographic system to video goggles we propose using the configuration shown in FIG. IB. Here, the crystalline lens of the eye is used to Fourier transform the modulated optical field, and the resulting image is projected directly onto the retina. Linearly polarized light is reflected off a polarizing beam-splitter prior to propagating through a quarter wave- plate. It is then reflected off a polarization- insensitive SLM, which modulates the wavefront. The quarter wave-plate makes the light linearly polarized again so that it is transmitted by the beam-splitter and enters the eye 102. An image is formed at photoreceptor 104, which can be the retina or a retinal prosthesis. A polarization-maintaining fiber can allow convenient delivery of optical power through a flexible cable from a light source (laser diode) located along with a battery and a pocket computer in an external unit. The lack of an intermediate image plane in this compact configuration makes it impossible to physically block undesired diffraction orders, therefore the effects of the undesired diffraction orders need to be minimized another way, as described in the next section. B2.2. Defocusing of the zero order

With a well-aligned nematic SLM such as the one used in this study, only the zeroth and first orders carry a significant fraction of the light power, so the effects of the higher diffraction orders can often be ignored for imaging purposes. The situation would be different if a ferroelectric SLM were used, for which there is an equal amount of power sent to the first positive and the first negative diffraction orders, so at the least the zeroth and the first negative order would need to be blocked in the intermediary image plane.

With the geometry of FIG. IB, there is no intermediary image plane, and therefore blocking these diffraction orders with an aperture is not possible. Whilst careful alignment and proper computation of the hologram can make the higher diffraction orders negligible compared to the first, the presence of a zero diffraction order is still unavoidable. With the nematic SLM we used (Boulder Nonlinear Systems, model P512- 0785), 77% of the light was diffracted into the first order. Higher efficiency SLMs can increase this fraction to above 90%. Because ferroelectric SLMs divert equal amounts of energy into the first positive and negative diffraction orders, they are not well suited to this approach, because over half of the incident light power would be lost.

When the SLM is illuminated by a collimated beam, the zero order appears as a bright focused spot in the image plane. With the power levels required for retinal prosthetic applications, tens of milliwatts incident on the SLM, the presence of a highly focused zero order raises safety concerns. To address this issue, we illuminate the SLM with a non- collimated beam which may be either diverging or converging, and refocus it by adding a suitable quadratic modulation on the SLM, which acts as a Fresnel lens.

FIGs. 2A-D illustrate this approach. When the SLM is turned off, all the light goes to the zero order, which is focused either behind the image plane with a diverging beam (FIG. 2A), or in front of it with a converging beam (FIG. 2B). When the SLM is turned on and a hologram is displayed (+ quadratic modulation for focusing the first order is applied) on it, the first order diffraction (solid lines) can be collimated to display an image in the focal plane, both with diverging (FIG. 2C) and converging beams

(FIG. 2D), while the de focused zero order (dotted lines) appears as a uniform

background. The effect of the zero order background on image quality is discussed below in section B3. The amount of beam divergence that can be compensated by the SLM is limited by its finite pixel pitch. With a maximum phase shift of 2π, a pixel size of 15 μιη, SLM resolution of 512 x 512 pixels and an illumination wavelength of 880 nm, the minimum focal length of the Fresnel lens that can be formed without exceeding the Nyquist sampling frequency in the spatial domain is—110 mm. Further increase of the diffraction angle would create aliasing effects producing shifted replicas of the same lens, which would significantly deteriorate the diffraction efficiency and therefore should be avoided. Since this aliasing limitation is determined by the pixel pitch of the SLM, it is common to both concave and convex Fresnel lenses, and thus affects to the same extent convergent and divergent illuminations of the SLM.

B2.3. Hologram computation

Algorithms for hologram synthesis have been studied extensively in the past. For our particular application, the algorithm must satisfy two main requirements. First, for testing visual acuity in animals, ID and 2D grating patterns with large uniform areas will be projected onto the retina, so the algorithm should produce images with good uniformity. Second, holograms should be computed fast enough to provide good substitution for natural vision, with refresh rates likely to exceed 15 Hz.

For our initial studies we selected the Gerchberg-Saxton (GS) algorithm, as it is well established, simple to implement, provides good uniformity over the image, and can be computed efficiently using GPU implementations of the fast Fourier transform. We implemented a CPU-based version of the GS algorithm in NI Lab VIEW 2012, running on a HP Z420 Workstation. It has been shown that this algorithm allows computation of the holograms at refresh speeds far exceeding video-rates, using a nVidia graphic cards in the OpenGL or CUDA programming environment. With the emergence of the mobile gaming industry, similar capabilities are being developed for mobile devices. Apple Inc. already offers a GPU-accelerated version of the 2D FFT on its iOS mobile operating systems, which would enable real-time computation of holograms on a mobile platform. B3. Performance analysis

B 3.1. Evaluation of image quality

Lasers are an excellent source of bright illumination, which often result in speckling due to high coherence. LCD and DMD-based systems can use low-coherence, high-power laser diode bars which greatly reduce this phenomenon by coupling multiple mutually-incoherent sources into the same fiber. Unfortunately, this is not an option for SLM-based imaging where high coherence is required. In principle, speckling can be reduced by incorporating a phase-smoothing constraint into the GS algorithm. However, the added computational complexity renders this approach impractical for real-time synthesis of holographic modulations.

Alternatively, speckles can be rapidly shifted and thereby smeared over time by sequentially displaying a series of holograms corresponding to different realizations of the same image. Each hologram of the series can be designed using the GS algorithm with different initial conditions (phase distributions). Rapid alternation of the holograms creating different speckling patterns efficiently blurs the speckles, and the image appears uniform to the observer, but computation of several different holograms for each image increases computational load on the system. Another approach to reduction of speckling is shift-averaging of the holograms. However, all these time-averaging methods can help only in systems having a sufficiently slow response time (e.g. normal photoreceptors, or slow-responding channelrhodopsin in optogenetic restoration of sight) compared to the refresh rate of the display.

In the case of photovoltaic retinal prosthesis having several diodes connected in series in each pixel the time- averaging approach is not applicable since the response time of photodiodes is orders of magnitude shorter than that of the fastest displays available (a few kHz with ferroelectric SLMs). In these devices, the current generated by each photodiode corresponds to the total light power absorbed. With three photodiodes connected in series in a pixel, the current delivered by each pixel corresponds to the minimum current generated by one of the three diodes: should any one diode in a pixel be shadowed, the entire pixel would be turned off. Since spatial integration instantaneously averages speckling over the photodiode surface it can greatly diminish speckling noise, so long as the speckle size is much smaller than a photodiode. We designed the optical pathway to generate images with speckles two-three times smaller than the width of the photodiodes (corresponding to four-nine speckles per diode area). To assess the effect of speckle integration over diodes of various sizes we acquired images of the projected grating patterns on a CCD camera and simulated the integrated response of the prosthesis to these speckled patterns. The image size was 3 mm x 3 mm, corresponding to the anticipated size of the tiled retinal implants, and a hexagonal array of 70 μιη pixels composed of three photodiodes each was tested. With larger pixels, the speckling noise should decrease in proportion to the diode area. The results showed that the dominant spatial frequency of the checkers was faithfully rendered, illustrating that speckles integrated over the photodiodes do not interfere with device output. As mentioned above, due to the requirement on compactness and simplicity of the imaging pathway, our geometry does not allow spatial filtering of the unwanted diffraction orders, particularly the zero order, which unavoidably carries a significant fraction of the optical power. Defocusing of the zero order in our system adds a diffuse background illumination over the whole image, thereby decreasing its contrast. FIG. 3A shows an image with the SLM turned off, where all of the optical power is directed to the zero order. FIG. 3B shows a checkerboard pattern projected with the first diffraction order superimposed with the residual defocused zero order.

With a non-zero background illumination, contrast of the image depends on how bright and therefore how sparse the image in the first order is. For grating patterns with 50% white and 50% black areas such as the one in FIG. 3B, we obtained contrast of 10: 1 within an image area of 1.1 x 1.1 mm . With sparser patterns, white parts become brighter and contrast improves, reaching values in excess of 100:1. Using a higher efficiency SLM would decrease the brightness of the zero order, thereby further increasing the contrast accordingly. For clinical uses of retinal implants, it is likely that information will be displayed using sparse contour images, as the ability of patients to recognize objects in this case is improved. It is therefore realistic to operate under the assumption of a certain degree of sparsity in the image displayed, which helps improve the maximum attainable contrast and has implications on power efficiency and safety of the system (see sections B3.3 and B3.4). Another potentially detrimental effect of liquid crystal displays is that during transition from one frame to another the state of the display is not well defined, and most of the light is deflected randomly. During this transition the first frame image fades, the defocused zero order appears in the center, and then disappears as the new frame image emerges.

With continuous illumination, transitions between successive holograms appear as bright flickering flashes of light across the visual field. Rapidly alternating between two different holograms corresponding to the same image makes this flickering appear to the naked eye as a constant background, thereby eliminating a distracting blink between the frames. However, this comes again at the cost of a reduced perceived contrast. Because of the capacitive nature of the coupling between implant and retina, optoelectronic retinal prostheses require pulsed illumination, therefore random transitions between the frames should not affect the system performance, as long as pulses of light are applied when the images are properly displayed.

B3.2. Field of view

Assuming a 2π maximum phase modulation per pixel, the addressable area on the image plane is determined by the wavelength and the SLM pixel size. With our d = 15 μιη pixel pitch and λ = 880 nm illumination, the maximum deflection angle that the SLM can provide is a = ± tan-1 (λ/d ) = ±3.4° . In a human eye having a focal length of 17 mm, this angular range corresponds to about a 2 x 2 mm addressable area on the retina, with a corresponding 10° diagonal field of view. A smaller pixel pitch would increase the field of view. For example, with a 8 μιη pixel pitch, commercially available on the market today, the maximum deflection angle of 6.3° will provide an addressable area of 3.7 x 3.7 mm on the retina, corresponding to 17 diagonal field of view. Since the visual field provided by a SLM-based display depends only on the amount of tilt applied to the incident waveform and the useful fraction of the beam reflected off the display corresponding to the first order can always be kept collimated, increasing the field of view does not change the efficiency of the display. It is important to note that diffraction efficiency can degrade significantly (by up to 60%) at the edges of the addressable square, compared to the center value. Therefore, SLMs with smaller pixel pitch should reduce the areas of reduced efficiency in the corners. With LCD-based systems the field of view is determined by the size of the display and magnification of the imaging optics used to project it onto the retina. For example, the Vuzix Smart Glasses Ml 00 has 16° diagonal field of view. Having a large field of view and retaining high efficiency is challenging, since very significant clipping of a wide beam occurs on the iris. A Kohler illumination system, in which the

illumination beam is focused onto pupil, could be used to minimize the losses.

B3.3. Power efficiency of the system

Power efficiency is naturally a very important consideration for a mobile system. Fully optical approaches to restoration of sight to the blind (i.e. without power supply connected to the retinal implant) require high irradiance levels to elicit neural response, which makes the power efficiency even more of a concern.

With a typical dynamic range of an image extending two orders of magnitude (2 log units, or contrast of white to black 100: 1), and assuming that, on average, images are in the middle of this range, i.e. having average luminance ten times below the maximum and ten times above the minimum, conventional displays (LCD and DMD) attenuate 90% of the illumination power. SLM-based holographic systems offer an excellent solution to this issue, because unlike LCD or DMD-based approaches, they do not waste optical power by attenuating the illuminating beam in dark areas, but rather redirect it into brighter parts of the image, thereby decreasing power requirements for the illuminating beam by an order of magnitude for a typical image. It is however important to note that for the system to reach its maximum efficiency, a polarization-insensitive SLM is required. Otherwise, a non- polarizing beam-splitter has to be used instead of the polarizing one shown in FIG. IB, which will lead to a 75% power loss.

The difference in efficiency becomes especially large when sparse patterns are projected, which is likely to be the case in clinical use of a retinal prosthesis. For a pattern which is 1% white and 99% black, the power consumption of the light source used to illuminate the SLM is just 1% of the power required for a LCD system. This difference becomes even larger when sequential activation schemes are used to stimulate the implant. As explained in section B3.4, if for safety reasons each frame displayed by the device needs to be broken into k subframes, LCD or DMD-based devices will require k times more energy to activate the implant compared to the single frame activation scheme, because the full beam power has to be used for each of the k subframes. In comparison, SLM-based devices can use Ilk of the full power in each of the subframes and the sequential activation scheme does not require more energy than the single frame activation scheme.

Redistribution of the incident optical power into the desired image in SLM systems also helps increase the irradiance of the image to much above the level of the display illumination, unlike LCD or DMD-based systems where the image irradiance cannot exceed irradiance of the display divided by the square of the magnification. For example, assuming equal image size, in a sparse black-and-white image (e.g. contour) having only 1% non-black pixels the illuminated pixels will be 100 times brighter using a SLM compared to an LCD. Brightness control in SLM systems necessitates an active control of the illumination source which changes its power from frame to frame according to the total amount of light in each image. The LCD or DMD systems however typically do not require adjustment of the light source from frame to frame since they attenuate the excessive light intensity in each pixel and white levels stay constant from frame to frame. Such redistribution of the illuminating light allows the use of

significantly lower power light sources, which helps not only to improve battery lifetime and compactness of the system, but also to reduce safety concerns regarding absorption of diffused light by the ocular tissues, as discussed below.

Another significant difference between SLM and LCD- based geometries in terms of power efficiency is the fact that with an SLM, the eye is always illuminated with a collimated or nearly collimated beam, regardless of the field of view of the display.

Therefore, a beam diameter which matches the pupil size will maximize the transmission efficiency. LCD screens, however, typically use non collimated illumination, which allows viewing of the full width of the display. With a diverging illumination and a field of view of Θ, the maximum fraction of light transmitted through the pupil is proportional to 1/ tan Θ . Thus, increasing the visual field from 10° to 20° will reduce transmission by at least 52%. These power losses can be minimized if a Kohler illumination layout (or other similar design) is used, in which illumination is focused on the iris plane into a spot smaller than the pupil diameter. However, eye movements restrict the applicability of such systems to lateral shifts of an eye in the iris plane that do not exceed the pupil radius. B3.4. Ocular safety considerations

All-optical approaches to restoration of sight to the blind require very bright illumination in order to elicit responses from the target neurons in the retina. While stimulation thresholds for optogenetic, photovoltaic polymer or photovoltaic silicon approaches are typically a few orders of magnitude below the maximum permissible exposure for the wavelength used, ocular safety limits the range of brightness

modulation. For the following discussion we will assume a system with 17° diagonal field of view, corresponding to approximately 3.7 x 3.7 mm (with a 5.2 mm diagonal) on the retina for a 17 mm focal length eye. We will consider an example of a 905 nm NIR wavelength, for which the average retinal irradiance limit is 5.3 mWmm (the peak irradiance limit is much higher and depends on pulse duration), but the reasoning can be applied to any other wavelength used for stimulation.

Since with LCD-based display the irradiance in each part of the image can only be attenuated compared to the uniformly illuminated white field, illumination of the LCD screen should be sufficient for supporting delivery of 5.3 mW mm over the whole 5.2 mm diagonal image. The entirety of this power has to fit through the pupil. With invisible near-IR illumination and dark goggles the pupil will likely dilate to at least 3 mm in diameter, while the use of bright visible illumination in patients having light sensitivity will likely result in pupil constriction below 2 mm. Therefore, the NIR irradiance in the pupil plane can reach values in excess of 10 mW mm . Even if the beam would fit within the pupil, natural eye movements will result in occasional exposures of the iris to potentially high irradiance levels. To reduce the total power delivered to the eye an eye tracking system could be used, which would turn off the parts of the screen that the user is not looking at. However, if the electronics of the display fail and the whole screen turns to white this tracking mechanism cannot prevent exposure of the iris to potentially unsafe irradiance levels. In the case of UV light (used for photoswitches), limitations are likely to apply for the cornea and crystalline lens exposures as well.

For SLM -based displays however, the entirety of the beam power can be concentrated in the image plane in a diffraction- limited spot, thereby providing very high local irradiance with very small beam power. For example, if a certain degree of sparsity in the image displayed is enforced, high irradiances can still be reached on the target image plane while reducing risks of damage on the iris. Should a 90% degree of sparsity be chosen, the total power in the beam required to display the image is then ten times lower than with a LCD or DMD-based display. In the previous example, this would correspond to irradiances on the pupil plane of 1 mWmm , compared to 10 mW mm with LCD or DMD displays. Additionally, sequential activation of the implant can help further reduce the requirements on total beam power. The SLM used in this study has a maximum image refresh rate of 203 Hz, but images will be refreshed much slower on the implant.

Assuming a target display refresh rate of 20 Hz, each frame displayed can be divided into up to ten distinct subframes, each used to address 1/lOth of the implant. In each subframe, at most 10% of the image is non-zero. With LCD or DMD- based displays, this sequential activation scheme does not translate into a reduction in maximum beam power required. It instead leads to a 10x decrease in power efficiency of the system, because the same amount of light has to be delivered to each subframe compared to the single image. However, with a holographic display, each subframe now requires 10x less beam power compared to the single frame: no additional power is required to use this sequential activation scheme, and the peak beam power can be reduced by a factor of 20. In the case of a failure of the display control, when the whole display turns white, and assuming the settings described in the previous example, this corresponds to irradiances of 1 mW mm on the pupil plane, compared to 10 mW mm with LCD or DMD displays.

Sequential activation of the implant and enforcing sparsity in the desired image can be combined in order to even further reduce requirements on total beam power.

Such use of power is inherently much safer than with the LCD screen-based display, since even with a failure of the screen control the power delivered to the eye remains within safe limits on the iris plane. Because the beam used to illuminate the eye is not collimated, the zeroth diffraction order cannot appear in focus on the retinal pigment epithelium and choroid either, which means that the incident light power is spread over an area corresponding to the full area addressable by the display on these tissues (for example, 3.7 x 3.7 mm for an 8 μιη pixel pitch SLM), making it inherently safe.

B4. In vivo testing of the system B4.1. VEP recording setup

As a proof of concept, the holographic display was used to evaluate the visual acuity of Long Evans (LE) rats in vivo. It should be noted that visual acuity is a complex visual function which is dependent on many parameters such as eye refractive power, optical quality of the image on the retina, cortical state, etc. In this paper, we evaluated the visual acuity by patterned visual evoked potentials (VEP) method, similar to that used in babies.

A projection system schematically as in FIG. 1, except that the SLM was illuminated at an angle and not through a beam-splitter and quarter-wave plate, was mounted on top of a slit-lamp (Zeiss, SL-120). The 638 nm illumination was provided by a 100 mW single-mode laser diode coupled into a single-mode fiber. The polarization of the beam was aligned to the preferred polarization axis of the SLM using a net zero order half-wave plate. Optical distortions in the system, especially those due to the SLM, are compensated using a wavefront correction algorithm. This algorithm can be implemented on the SLM itself and does not require extra hardware. A tandem of lenses (/= -100 mm and = 60 mm) was used to project the image onto the focal plane of the slit-lamp. The non-polarizing beam-splitter lets the user visually align the system. To allow direct observation of the projected light patterns on the retina via slit-lamp the focusing power of the cornea in the animal eye was cancelled using a cover slip and viscoelastic material (sodium hyaluronate, Pro Vise, Alcon Laboratories).

To record the VEP, three screw electrodes were inserted into the skull and secured with cyanoacrylate glue and dental acrylic. Two electrodes were placed over the visual cortex of both hemispheres, 4 mm lateral from midline, 6 mm caudal to the bregma. One reference electrode was implanted 2 mm right to the midline and 2 mm anterior to the bregma. Nose and tail needle electrodes served as a reference and the ground, respectively. The signals were recorded using a Diagnosys Espion E3 VEP system, using the built-in pre-amplifiers, amplifiers and line filters. All signals were band-pass filtered between 1.25 and 500 Hz.

Rats were anaesthetized with a mixture of Ketamine (75 mg kg-1 ) and Xylazine (5 mg kg-1 ) injected intra-muscularly. Additional injections of 50% of the initial dose were administered every 45 min, or as needed. A heating pad was used to maintain the body temperature at 37.5 ± 0.5 °C. Electrophysiological recordings were conducted with a dim room illumination of 250 nW cm . All animal care and experiments were carried out in accordance with the ARVO guidelines for the Use of Animals in Ophthalmic and Vision Research and approved by the Stanford Administrative Panel on Laboratory Animal Care. Following pupil dilation and application of the cover slip, the 638 nm laser beam was directed into the eye and the image was visually aligned via slit-lamp, to appear in focus on the retina.

B4.2. Effect of hologram swapping on VEP recordings To assess visual acuity of the animal, one-dimensional black- and- white gratings were projected onto the retina, and the square wave contrast was reversed (black-to-white and back) with a frequency of 2 Hz. Amplitude of the steady- state VEP signal at this frequency was used to assess the cortical response. VEP signals decrease with increasing spatial frequency of the patterns, and it reaches the noise level at the limit of visual acuity of the animal. We first illustrate the effect of the random light redistribution during hologram transitions on VEP recordings. With continuous illumination, this effect resulted in flashes of light at the contrast alternation rate of 2 Hz. This artifact obscured the cortical response to alternations of the stripe positions on the retina

FIGs. 4A-B show measured cortical responses (VEP) to optical stimuli projected onto the retina. FIG. 4A shows the average of 1000 recordings taken with 2 Hz contrast reversal of two gratings having a very high density of 32 stripes mrrT1 on the retina, significantly exceeding the visual acuity limit of rats. Despite the fact that rats cannot resolve such a dense pattern, there is a clear response at t = 170 ms and t = 670 ms, illustrating the effect of flickering caused by the random light redistribution during hologram transitions. The large number of trials required to obtain a signal is explained by the fact that only a small area (1.2 x 1.2 mm ) of the retina was illuminated by the pattern, resulting in a weak VEP signal.

To eliminate this artifact we presented two different holograms for each of the two complementary patterns, alternating at 40 Hz, while the contrast reversal rate remained at 2 Hz, as described in section B3.1. The method described in this section was initially presented to address the speckling problem associated with coherent illumination, but can also be used to reduce the effect of the random light redistribution, as we illustrate here. Flickering at 40 Hz did not induce any noticeable VEP signal, while the contrast reversal at 2 Hz retained a clear response, as illustrated in FIG. 4B. Here FIG. 4B shows that VEP recordings for gratings with 32 stripes mrrT1 on the retina did not produce any signal, while patterns with 8 stripes mrrT1 show a clear signal at 2Hz (t= 160ms and t=660 ms). As before, 1000 trials were averaged for this demonstration.

B5. Discussion

The holographic goggles system presented here has appealing features compared to more traditional LCD or DMD-based systems for activation of photovoltaic retinal prosthesis or other approaches to restoration of sight that require very bright illumination. We demonstrate that despite the presence of speckles and the zero diffraction order background, it is possible to obtain contrast of 10: 1 for images having 50% white and 50% black, and over 100: 1 for sparse contour images. The problem of the random light redistribution during hologram transitions can be overcome by high frequency exchange of alternative versions of the holograms encoding the same images. Using this technique, we demonstrated cortical response to motion in rats. However, in applications requiring short-pulse illumination, such as photovoltaic array, proper synchronization of the pulse of light with the display refresh timing will eliminate this problem altogether.

The high brightness requirements for optical approaches to restoration of sight presents at least two significant challenges: ocular safety and battery lifetime, both of which can be addressed using SLM-based display. While LCD or DMD should be illuminated at the maximum expected brightness over the entire area, the SLM can redistribute light into areas as small as a diffraction-limited spot. Therefore, requirements on total beam power can be reduced by orders of magnitude by enforcing a certain degree of sparseness in the patterns projected or using sequential activation schemes, making this approach inherently safer, especially in the event of a failure which would lead to projection of a white screen. Reduction of the average power also translates into a corresponding extension of the battery life.

Holographic displays, however, also suffer from a number of drawbacks.

Miniaturizing a traditional Fourier holographic system is challenging from an engineering perspective. The layout presented here is a possible solution to this technical difficulty, but the price paid is in the presence of a zero diffraction order in the final image which cannot be spatially filtered out. However, when a high-efficiency SLM is used, this zero diffraction order does not impact image quality significantly, especially for sparse patterns. Real-time computation of the holograms presents a significant challenge. While current desktop computers are capable of it, real- time computation of holograms using the GS algorithm on mobile platforms requires GPU-acceleration which is just becoming available with the iOS mobile platform (Apple Inc.). SLM-based systems also suffer from speckling, and the time-averaging methods traditionally used to address this problem do not work in the context of photovoltaic retinal prosthesis due to their very high response speed. Spatial integration of speckles over the size of a photodiode helps reducing this noise, as long as the speckles are significantly smaller than photodiodes. Using the diffraction limit as an estimate of speckle size, speckles in the image plane of our system are about 5-8 μιη, corresponding to about six speckles per the smallest diode (20 um) used in the photovoltaic implants. Direct optical stimulation of single neurons in the retina has much slower response time than photodiodes, and therefore speckles can be averaged over time, using techniques such as shift- averaging. Unlike SLM, the LCD or DMD based systems can use low- coherence illumination sources, such as high- power laser diode bars, resulting in much less-speckled images.

Unlike LCD or DMD-based systems that attenuate illumination in each pixel to form an image, the SLM-based imaging also requires an active control of the light source to adjust the total power for each frame, corresponding to the average brightness of the image.

Several improvements or modifications to this system could be envisioned.

Throughout our study, we encoded the Fresnel lens used to collimate the first order over a 2π modulation. Use of a larger phase shift on each pixel (4π or more, available with some SLM models) would allow further defocusing of the zero order and increasing the visual angle: 4π modulation per pixel would allow twice steeper deflection, which enables collimation of a twice more diverging beam. The zero order could therefore be spread over an area 4 times larger, improving the contrast accordingly. This approach, however, might be accompanied by generation of stronger higher orders along the optical axis, which may interfere with the image.

SLMs also allow correcting systematic aberrations in the optical system, which would otherwise degrade the image quality. This feature could be used to compensate for some aberrations in the patient's eye, such as astigmatism, near or far-sightedness, and even early stages of cataract. This would require measuring the aberrations of the eye and correcting for them in the computed hologram. B6. Conclusions

Holographic imaging systems designed for safe and efficient activation of photovoltaic retinal prosthesis enable the projection of contour images with high efficiency, high irradiance and much lower total power than traditional LCD or DMD- based displays. Integration of light over the photosensitive elements reduces speckling noise to acceptable levels for diodes as small as 20 μιη. Very compact design of video goggles is based on defocusing of the zero diffraction order, and refocusing the image using Fresnel lens added to the hologram of the encoded image. Solutions to various challenges associated with the holographic approach, such as the presence of multiple diffraction orders, speckles, transitions between the holograms and difficulties in hologram computation are provided. As a proof of concept, the system was successfully tested in vivo by measuring cortical responses to alternating gratings, thus demonstrating feasibility of the holographic approach to near-the-eye display.

Claims

1. A method for providing an image to an eye of a patient, the method comprising: receiving an input image; computing a hologram corresponding to the input image; encoding the hologram on a phase-only spatial light modulator (SLM); illuminating the SLM with an optical beam such that phase modulated light from the SLM is received by the eye; wherein an output image corresponding to the input image is formed in the eye; wherein focusing of the optical beam is configured such that zero order light from the SLM is out of focus at a location of the output image.
2. The method of claim 1 , wherein the eye includes a photoreceptor disposed at a photoreceptor location, and wherein the SLM is disposed at a Fourier conjugate location of the photoreceptor location.
3. The method of claim 2, wherein the SLM has npx pixels and a width w in its largest dimension, wherein λ is a wavelength of the optical beam, wherein (|)max is a maximum per pixel phase shift of the SLM, wherein Imax is a maximum permitted optical irradiance, wherein f is a focal length of the eye, wherein Pi is a power of the optical beam as received by the eye,
-1
wherein 6L _ = tan
2π ν
Figure imgf000033_0001
wherein the optical beam has a focusing angle Θ such that 0mm < |θ| < Θ ',max-
4. The method of claim 1, wherein the output of the SLM is substantially in the zero order and a single non-zero diffraction order.
5. The method of claim 1, wherein the SLM comprises a nematic liquid crystal spatial light modulator configured to control the distribution of incident light among output diffraction orders.
6. The method of claim 1, wherein a lateral size of the optical beam is configured to match an iris of the eye.
7. The method of claim 1, further comprising providing a quadratic phase shift with the SLM in order to make the output image three-dimensional.
8. The method of claim 1, further comprising performing speckle removal.
9. The method of claim 8, wherein the speckle removal comprises a method selected from the group consisting of: modulating a source of the optical beam, and using multiple holograms for each input image.
10. The method of claim 1, further comprising a frame switching method selected from the group consisting of: initializing computation of a current frame hologram using a hologram from a previous frame; synchronizing a source of the optical beam and the SLM such that the SLM is not illuminated during frame transitions, and providing a frame rate greater than 20 frames per second.
11. The method of claim 1 , wherein the output image is disposed at a retinal prosthesis within the eye.
12. The method of claim 1, wherein the computing a hologram corresponding to the input image comprises recognizing relevant features in the input image according to a computer vision method, and computing a hologram corresponding to the recognized relevant features.
PCT/US2014/026603 2013-03-14 2014-03-13 Holographic head-mounted display WO2014151877A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US201361783170P true 2013-03-14 2013-03-14
US61/783,170 2013-03-14

Publications (1)

Publication Number Publication Date
WO2014151877A1 true WO2014151877A1 (en) 2014-09-25

Family

ID=51581058

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/026603 WO2014151877A1 (en) 2013-03-14 2014-03-13 Holographic head-mounted display

Country Status (1)

Country Link
WO (1) WO2014151877A1 (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017062483A1 (en) * 2015-10-05 2017-04-13 Magic Leap, Inc. Microlens collimator for scanning optical fiber in virtual/augmented reality system
WO2017127547A1 (en) * 2016-01-20 2017-07-27 Magic Leap, Inc. Polarizing maintaining optical fiber in virtual/augmented reality system
US9798147B1 (en) 2015-05-28 2017-10-24 Verily Life Sciences Llc Near-eye display with phase map
US9810910B1 (en) 2015-05-28 2017-11-07 Verily Life Sciences Llc Contact lens with phase map display
US10048647B2 (en) 2014-03-27 2018-08-14 Microsoft Technology Licensing, Llc Optical waveguide including spatially-varying volume hologram
US10092177B1 (en) 2015-12-30 2018-10-09 Verily Life Sciences Llc Device, system and method for image display with a programmable phase map
US10180769B1 (en) 2016-04-12 2019-01-15 Google Llc Symbol display
US10210844B2 (en) 2015-06-29 2019-02-19 Microsoft Technology Licensing, Llc Holographic near-eye display
US10254536B2 (en) 2015-07-20 2019-04-09 Magic Leap, Inc. Collimating fiber scanner design with inward pointing angles in virtual/augmented reality system
US10254542B2 (en) 2016-11-01 2019-04-09 Microsoft Technology Licensing, Llc Holographic projector for a waveguide display
US10317690B2 (en) 2014-01-31 2019-06-11 Magic Leap, Inc. Multi-focal display system and method
US10338391B2 (en) 2015-10-06 2019-07-02 Magic Leap, Inc. Virtual/augmented reality system having reverse angle diffraction grating
EP3583886A1 (en) * 2018-06-19 2019-12-25 Nokia Technologies Oy Accurate optical devices for age-related macular degeneration
US10545337B2 (en) 2015-10-28 2020-01-28 Samsung Electronics Co., Ltd. See-through holographic display apparatus

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10319240A (en) * 1997-05-22 1998-12-04 Fuji Xerox Co Ltd Head-mounted display
US20080212034A1 (en) * 2007-03-02 2008-09-04 Lucent Technologies Inc. Speckle reduction in laser-projector images
US20090002787A1 (en) * 2005-06-15 2009-01-01 Adrian Cable Holographic Display Devices
JP2010026273A (en) * 2008-07-22 2010-02-04 Dainippon Printing Co Ltd Head mounted display
US20100238528A1 (en) * 2006-06-09 2010-09-23 Koninklijke Philips Electronics, N.V. Method and system for electroholographic display with zeroth-order diffraction suppression

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10319240A (en) * 1997-05-22 1998-12-04 Fuji Xerox Co Ltd Head-mounted display
US20090002787A1 (en) * 2005-06-15 2009-01-01 Adrian Cable Holographic Display Devices
US20100238528A1 (en) * 2006-06-09 2010-09-23 Koninklijke Philips Electronics, N.V. Method and system for electroholographic display with zeroth-order diffraction suppression
US20080212034A1 (en) * 2007-03-02 2008-09-04 Lucent Technologies Inc. Speckle reduction in laser-projector images
JP2010026273A (en) * 2008-07-22 2010-02-04 Dainippon Printing Co Ltd Head mounted display

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10317690B2 (en) 2014-01-31 2019-06-11 Magic Leap, Inc. Multi-focal display system and method
US10048647B2 (en) 2014-03-27 2018-08-14 Microsoft Technology Licensing, Llc Optical waveguide including spatially-varying volume hologram
US9798147B1 (en) 2015-05-28 2017-10-24 Verily Life Sciences Llc Near-eye display with phase map
US9810910B1 (en) 2015-05-28 2017-11-07 Verily Life Sciences Llc Contact lens with phase map display
US10210844B2 (en) 2015-06-29 2019-02-19 Microsoft Technology Licensing, Llc Holographic near-eye display
US10254536B2 (en) 2015-07-20 2019-04-09 Magic Leap, Inc. Collimating fiber scanner design with inward pointing angles in virtual/augmented reality system
WO2017062483A1 (en) * 2015-10-05 2017-04-13 Magic Leap, Inc. Microlens collimator for scanning optical fiber in virtual/augmented reality system
US10338391B2 (en) 2015-10-06 2019-07-02 Magic Leap, Inc. Virtual/augmented reality system having reverse angle diffraction grating
US10545337B2 (en) 2015-10-28 2020-01-28 Samsung Electronics Co., Ltd. See-through holographic display apparatus
US10092177B1 (en) 2015-12-30 2018-10-09 Verily Life Sciences Llc Device, system and method for image display with a programmable phase map
CN108474904A (en) * 2016-01-20 2018-08-31 奇跃公司 Polarization-maintaining fiber in virtually/augmented reality system
WO2017127547A1 (en) * 2016-01-20 2017-07-27 Magic Leap, Inc. Polarizing maintaining optical fiber in virtual/augmented reality system
US10180769B1 (en) 2016-04-12 2019-01-15 Google Llc Symbol display
US10254542B2 (en) 2016-11-01 2019-04-09 Microsoft Technology Licensing, Llc Holographic projector for a waveguide display
EP3583886A1 (en) * 2018-06-19 2019-12-25 Nokia Technologies Oy Accurate optical devices for age-related macular degeneration

Similar Documents

Publication Publication Date Title
US8113657B2 (en) Device and method for determining the orientation of an eye
EP2152143B1 (en) System and method for illumination and fixation with ophthalmic diagnostic instruments
US8488895B2 (en) Laser scanning digital camera with pupil periphery illumination and potential for multiply scattered light imaging
TWI297791B (en) Methods and apparatuses for altering relative curvature of field and positions of peripheral, off-axis focal positions
US8684527B2 (en) Ophthalmic diagnostic instrument
JP4384363B2 (en) Logarithmic photomultiplier for use with a photoreceptor-based transplanted retinal prosthesis and the prosthesis
US6124954A (en) Projection screen based on reconfigurable holographic optics for implementation in head-mounted displays
JP6449236B2 (en) Method and apparatus for a multiple exit pupil head mounted display
EP1183558B1 (en) Holographic displays
US6916096B2 (en) System and method for recording the retinal reflex image
US5653751A (en) Systems and methods for projecting an image onto a retina
US7502160B2 (en) Speckle reduction in laser-projector images
CN106662731B (en) Wearable 3D augmented reality display
DE60133656T2 (en) Performance adjustment for adjustable lens
CN105934902B (en) Virtual and augmented reality System and method for
Maimone et al. Computational augmented reality eyeglasses
JP2014503836A (en) Display device
Reichelt et al. Depth cues in human visual perception and their realization in 3D displays
US10345599B2 (en) Tile array for near-ocular display
US8220926B2 (en) Visual display with illuminators for gaze tracking
US20130286053A1 (en) Direct view augmented reality eyeglass-type display
JP2008509438A (en) Optical display device scanned with variable fixed viewing distance
JP3396062B2 (en) Image display device
US10379353B2 (en) Augmented and virtual reality display systems and methods for diagnosing health conditions based on visual fields
DE69534116T2 (en) Binocular laser raster ophthalmoscope

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14770571

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14770571

Country of ref document: EP

Kind code of ref document: A1