Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3141—Constructional details thereof
  • H04N9/315—Modulator illumination systems
  • H04N9/3161—Modulator illumination systems using laser light sources
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B26/00—Optical devices or arrangements for the control of light using movable or deformable optical elements
  • G02B26/06—Optical devices or arrangements for the control of light using movable or deformable optical elements for controlling the phase of light
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/0012—Optical design, e.g. procedures, algorithms, optimisation routines
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/18—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for optical projection, e.g. combination of mirror and condenser and objective
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/50—Optics for phase object visualisation
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B21/00—Projectors or projection-type viewers; Accessories therefor
  • G03B21/005—Projectors using an electronic spatial light modulator but not peculiar thereto
  • G03B21/006—Projectors using an electronic spatial light modulator but not peculiar thereto using LCD's
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B21/00—Projectors or projection-type viewers; Accessories therefor
  • G03B21/005—Projectors using an electronic spatial light modulator but not peculiar thereto
  • G03B21/008—Projectors using an electronic spatial light modulator but not peculiar thereto using micromirror devices
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3102—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM] using two-dimensional electronic spatial light modulators
  • H04N9/312—Driving therefor
  • H04N9/3126—Driving therefor for spatial light modulators in series
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3141—Constructional details thereof
  • H04N9/315—Modulator illumination systems
  • H04N9/3155—Modulator illumination systems for controlling the light source
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3141—Constructional details thereof
  • H04N9/315—Modulator illumination systems
  • H04N9/3167—Modulator illumination systems for polarizing the light beam
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3179—Video signal processing therefor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3179—Video signal processing therefor
  • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3179—Video signal processing therefor
  • H04N9/3188—Scale or resolution adjustment

Definitions

• This invention relates to generating desired patterns of light.
• the desired patterns of light correspond to images specified by image data.
• Specific embodiments provide methods for controlling a free- form lens such as a phase-shifting light modulator, a variable mirror, or the like to achieve a desired distribution of light.
• Other embodiments provide projectors for projecting light.
• HDR (high dynamic range) image projection may be achieved by providing two or more stages of light modulators (Hoskinson et al.).
• Many light modulators e.g. LCD panels
• subtraction i.e. by absorbing unwanted light.
• Some efforts have been made to create desired light fields by reallocating light.
• many available light reallocation technologies have significant disadvantages. For example, some require laser light, which can result in laser speckle. Some are very computationally intensive. Some require very high spatial frequency control of light which places demands on light modulators and also can result in artifacts caused by diffraction of light.
• Freeform lenses which can be aspherical, asymmetric lenses may be designed to generate specific caustic images under pre-defined illumination conditions [Finckh et al. 2010, Papa et al. 2011, Schwartzburg et al. 2014, Yue et al. 2014].
• the caustic image is a redistribution or "reallocation" of light incident on the freeform lens [Hoskinson et al. 2010]
• Computer graphics approaches to designing such freeform lenses are known as goal-based caustics. Designing a freeform lens to achieve a particular desired image can be computationally intensive.
• Freeform lenses may be applied for general lighting applications (e.g. [Minano et al. 2009]) and more specifically for goal-based caustics [Berry 2006, Hullin et al. 2013].
• Some methods for designing freeform lenses apply discrete optimization methods that work on a pixelated version of the problem (e.g. [Papas et al. 2011 , Papas et al. 2012, Papas et al. 2012]).
• Others optimize for continuous surfaces without obvious pixel structures (e.g. [Finckh et al. 2010, Kiser et al. 2013, Pauly and Kiser 2012, Schwartzburg et al. 2014, Yue et al. 2014]).
• Holographic image formation models (e.g. [Lesem et al. 1969]) have been adapted to create digital holograms [Haugen et al. 1983].
• Holographic projection systems have been proposed for research and specialty applications [Buckley 2008]. Many of these systems use diffraction patterns (or holograms) addressed on a phase SLMs in combination with coherent light (lasers) for image generation.
• Holographic projection generally requires coherent light.
• the inventors have recognized a need for more efficient ways to design freeform lenses to achieve desired light patterns.
• the inventors have determined that sufficiently efficient design methods may be applied to provide real-time or near real time generation of dynamic freeform lenses.
• Such dynamic freeform lenses may, for example deliver video content or dynamically-changing light effects.
• This invention provides methods for controlling spatial light modulators to provide free-form lensing of light.
• the light may be projected and/or further modulated.
• Another aspect of the invention provides apparatus such as projectors, displays, illumination systems and their components that implement methods as described herein.
• Dynamic freeform lenses may be applied in light projection systems. Such light projection systems may advantageously be light efficient, provide high (local) peak luminance, and high contrast (high dynamic range, HDR).
• Some embodiments employ a dynamic freeform lens, implemented on a phase only SLM.
• the phase only SLM may be combined with a conventional light blocking SLM such as a reflective LCD in a cascaded modulation approach.
• a phase modulator can create a smooth, but still quite detailed "caustic" image.
• Such a caustic image may be further modulated by an amplitude modulator if so desired.
• This approach may provide both a higher dynamic range and/or improved (local) peak luminance as compared to conventional projectors.
• phase modulator • illumination systems and projectors in which a phase modulator is illuminated with (near-)collimated light and a phase pattern addressed on the phase modulator forms an image or desired light field with or without further optical elements;
• a dual-modulation projector design that combines a phase modulator and an amplitude modulator for image generation and is capable of working with broadband light as well as monochromatic light (such as laser light).
• phase modulation function can be solved for in the in the lens plane instead of the image plane, for example using optimization methods, to arrive at a very simple to implement method that optimizes directly for the phase function or the shape of a refractive lens, without requiring additional steps.
• This approach may be solved very efficiently in the Fourier domain. In some embodiments the algorithm is efficient enough for on-the fly
• One example aspect provides a dual-modulation projector design, in which one spatial light modulator that affects only the phase of the illumination is combined with one spatial light modulator that affects its amplitude (intensity).
• the phase-only modulator curves the wavefront of light reflected off it, and acts as a pre-modulator for a
• the dual-modulation HDR projector design uses the freeform lens optimization approach to provide energy efficient high dynamic range and high intensity projection. This approach is capable of using white light (or other broadband light) illumination as well as coherent laser light. Use of broadband light can yield a significant improvement in image quality by eliminating laser speckle and averaging out other diffraction artifacts.
• a real-time implementation of a high resolution freeform lens enables applications such as video processing.
• a dual-modulation HDR projector may be constructed entirely from robust components that are currently commercially available.
• the phase modulator creates a smoothed, but still quite detailed "caustic" image on the amplitude modulator. Since the caustic image merely redistributes, or “reallocates”, light this approach produces both a higher dynamic range as well as an improved (local) peak brightness, compared to conventional projectors that modulate light using a single amplitude modulator.
• Some embodiments apply a linear model in which the local deflection of light is proportional to the gradient of a phase modulation function, while the intensity is proportional to the Laplacian.
• Some embodiments combine application of this model with a parameterization of the optimization problem in the lens plane instead of the image plane to arrive at a very simple to implement method that optimizes directly for the phase function or the shape of a refractive lens, without any additional steps.
• the objective function is non- convex due to an image warping operator convergence can typically be achieved within a few iterations.
• Technology as described herein has application in controlling dynamic freeform lenses, for example in the context of light efficient, high (local) peak brightness, and high contrast (high dynamic range, HDR) projection systems.
• Some aspects of the invention provide algorithms which may be applied to efficiently determine phase patterns for a phase modulator to cause a desired light profile in the image plane.
• a (near) one-to-one relationship is established between the phase at a location in the lens plane and a corresponding area of the image plane. This is in contrast to the diverging or converging rays or beams that are required for traditional holographic approaches.
• Figure 1 is a schematic illustration of an example geometry for image formation. Phase modulation takes place in the lens plane, which is placed at a focal distance of / from the image plane. This results in a curvature of the wavefront, represented by a phase function p( ).
• Figure 2 is a schematic illustration showing intensity change due to the distortion of a differential area dx.
• Figure 3 is a schematic illustration showing a geometry for refraction in a freeform lens defined by a height field h(x).
• Figure 4 shows stages in an algorithm for freeform lensing.
• Figures 5C, 5D, and 5E show examples of refractive lenses produced using methods described herein.
• Figures 5A and 5B show a phase-only spatial light modulator being used to drive a projector display with white light. The same setup could also use laser illumination. This approach is particularly useful in energy-efficient dual modulation HDR projectors.
• the right hand image shows refractive lenses designed using the same free form lensing algorithm for goal-based caustics. For photography purposes, both results are shown on back-illuminated rather than front screens, so that the displayed 'Lena' image appears mirrored.
• Figures 6A and 6B are photographs of prototype embodiments.
• a narrow -band, dynamic lensing test setup comprising a HeNe laser source, a beam expander, a linear polarization filter and folding mirrors, the phase-only SLM and a projection screen at 50mm distance from the SLM.
• the SLM phase pattern used to generate the freeform lens (in this case the Siggraph logo) is also displayed on the notebook screen for visualization. Note the Fresnel-like phase wrapping used to achieve larger phase changes.
• the white light configuration bypasses the laser module, and comprises a white LED, collimation optics and linear polarization filter, the phase-only SLM and a projection screen at 50mm distance from the SLM.
• the SLM in this setup was calibrated for a center wavelength of 550nm.
• FIG. 7 is a system diagram of an example high brightness, HDR projector: light from an expanded and collimated laser beam is reflected off a phase-only modulator. The per-pixel amount of phase retardation resembles the height field of the dynamic lens calculated with an algorithm as described herein.
• the effective focal plane of this free form lens is in-plane with an off-the-shelf, reflective projection head consisting of the polarizing beam splitter together with an LCoS microdisplay and a projection lens. Light from dark parts of the image can be used to create high luminance features, and simultaneously reduce the black level.
• FIG 8 is a system diagram of an example high brightness, HDR projector including an intermediary image plane in which light from the phase stage can be further shaped, for example by adding a light shaping diffuser: light from an expanded and collimated laser beam is reflected off a phase-only modulator.
• the per-pixel amount of phase retardation resembles the height field of the dynamic lens calculated with an algorithm as described herein.
• the effective focal plane of this free form lens is in-plane with an intermediary image plane, which is relayed onto an off-the-shelf, reflective projection head comprising the polarizing beam splitter together with an LCoS microdisplay and a projection lens via relay optics.
• Light from dark parts of the image can be used to create high luminance features, and simultaneously reduce the black level.
• Phase Pattern the phase pattern as computed by Algorithm 1.
• Simulation Huygens-Fresnel simulation of predicted image.
• Direct photograph of actual image without diffuser showing diffraction artifacts.
• Diffuser by adding the a thin- film diffuser, artifacts such as diffraction fringes nearly completely mitigated.
• Standard photo of standard, amplitude modulation only projection using a single amplitude modulator shows elevated black levels and low contrast.
• Figures 10A, 10B, and IOC From left to right correlating to positions A to C in Figure 8: A: phase pattern present at phase-only LCoS modulator, B: a direct image produced by lens in intermediary image plane (prior to diffuser) and C: intensity distribution present at amplitude LCoS modulator after having passed through a thin-film light-shaping diffuser.
• Figures 11 A, 11B, and 11C show an example high-dynamic range projection system based on dual modulation.
• a first stage modulates the source illumination phase to form a coarse intermediate image. This is followed by an amplitude modulation stage that forms the final image.
• Using phase modulation results in greater contrast and darker black- levels than conventional projection since light is redistributed rather than blocked.
• Figure 12A shows geometry for the image formation model, with phase modulation p( ) taking place in the lens plane, and resulting deflections creating a caustic image on the image plane at distance /.
• Figure 12B shows the local intensity on the image plane is related to the change in the differential surface area between corresponding patches on the lens plane and the image plane.
• Figures 13A, 13B, and 13C By mirror-padding the input image, pure-Neumann boundary conditions at the image edge can be achieved while retaining a Toeplitz matrix structure. This prevents distortions of the image boundary. Simulated results with LuxRenderTM.
• Figures 14A, 14B, 14C, and 14D LuxRender raytracing simulations: the smoothness parameter a penalizes strong caustics in the image that achieve high- brightness but poor image quality.
• Figure 15 Layout of a simple example dynamic lensing test setup for use with broadband light.
• a beam of light from a light source such as a white LED with collimation optics (a modified flash light) together with a linear polarization filter (provided for sensible use of the phase modulator) is reflected off the SLM operated in phase-only mode and onto a small projection screen facing the SLM at a 50mm distance.
• the SLM in this setup was calibrated for a center wavelength of 550nm. Due to light-engine power limitations, this setup was not sufficient to drive a dual-modulation setup (the reduced intensity also introduces camera capture noise in the inlay) although it illustrates that phase modulation is functional with broadband light. This paves the way for future broadband illumination phase+amplitude dual-modulation setups.
• Such setups could apply industry standard Xenon bulbs, cost effective blue laser +phosphor light sources or LEDs, for example as light sources.
• Figure 16 Single modulation test setup for lasers comprising a light source (yellow box, 532nm DPSS laser and laser controller), beam expansion and collimation optics (orange box), the reflective phase SLM (blue), various folding mirrors and a simple projection lens to relay the image from and intermediate image plane onto the projection screen (green).
• the phase pattern shown on the computer screen correlates linearly to the desired phase retardation in the optical path to form the image. It has been phase- wrapped at multiples of one wavelength and can be addressed directly onto the micro display SLM.
• Figure 17 Simplified system diagram of an example high brightness, HDR projector: light from an expanded and collimated laser beam is reflected off a phase-only modulator. The per-pixel amount of phase retardation resembles the height field of the dynamic lens calculated with our algorithm. The effective focal plane of this freeform lens is in-plane with an off-the-shelf, reflective projection head consisting of the polarizing beam splitter together with an LCoS microdisplay and a projection lens. Light from dark parts of the image can be used to create high luminance features, and simultaneously reduce the black level.
• Figure 18 Comparison of simulated and captured results from top to bottom by row.
• Phase Pattern the phase pattern as computed by Algorithm 4.1. Simulation:
• Huygens-Fresnel simulation of predicted image Direct: photograph of actual image without diffuser showing diffraction artifacts. Diffuser: by adding the a thin-film diffuser, artifacts such as diffraction fringes nearly completely mitigated. Standard: photo of standard, amplitude modulation only projection using a single amplitude modulator shows elevated black levels and low contrast. Proposed (HDR): Using our lensing approach redistributes light from dark regions to bright regions, resulting in improved black levels and increased highlight intensity. The last two rows appear slightly distorted due to an off- angle position of the camera which became necessary because of a short throw projection and close screen as well as baffles to block ambient light effectively to capture the black level of the system.
• FIGS 19A and 19B Photos of a prototype projector in LDR comparison mode (left image) and HDR mode (right image). Left: light redistribution is active resulting in increased peak luminance and reduced black level. Right: LDR projector for comparison using the same hardware.
• LDR mode a flat phase profile results in a uniform illumination profile at the amplitude attenuator (second SLM). Each image is vertically split to show a long exposure time on the left half (dark level detail is visible) and a short exposure on the right side (detail in the highlights is visible). Both exposures are of the same projected image on screen.
• Some embodiments provide a new approach to determining a lens shape or phase function that can provide a desired light field when illuminated.
• the output of this approach may be applied to control a phase modulator or variable lens or variable mirror to yield the desired light field.
• phase-only SLMs are used as programmable freeform lenses.
• the lenses may be illuminated with broadband light (e.g. white light). This eliminates speckle, while at the same time the spatial smoothness of the lens modulation patterns reduces diffraction artifacts. Any remaining diffraction is averaged out by the broadband nature of the illumination, resulting only in a small amount of blur that can be modeled and compensated for in a dual-modulation setting
• Some embodiments optimize directly for the phase function, or, equivalently, the lens shape, without a need for a subsequent integration step. This is facilitated by a parameterization of the problem that expresses the optimization directly in the lens plane rather than the image plane. This leads to a much simpler formulation of the freeform lens optimization problem than the approaches described in the literature.
• This application relates in part to methods for displaying desired light patterns by using a modulator that does not absorb much light, but moves it around within the image plane.
• the modulator may be controlled to provide moving, bright spots of light.
• An example of a modulator suitable for this application is a LCoS SLM operated in a phase- only fashion.
• the SLM may have a suitable resolution such as 1, 2 5 or more megapixels. Control of the SLM may be achieved by optimizing a continuous phase function representing the required curvature of the wavefront of light as it passes through the SLM.
• Apparatus and methods according to different embodiments allow the use of broadband light (e.g. from a lamp, LEDs, or arrays of lasers with different wavelengths) as well as monochromatic laser light.
• Phase modulating arrays such as liquid crystal-based SLMs operated in a phase-only configuration are applied as programmable freeform lenses.
• Being able to use broadband illumination can help to eliminate screen speckle, while at the same time the spatial smoothness of the lens modulation patterns reduces other artifacts such as diffraction. Any remaining diffraction effects in the image plane can be averaged out by the broadband nature of the illumination, resulting only in a small amount of blur that can be easily modeled and compensated for by providing one or more additional modulators.
• phase function i.e. the shape of the wavefront in the lens plane
• lens shape without a need for a subsequent integration step
• a parameterization of the problem that allows us to express the optimization directly in the lens plane rather than the image plane.
• a lens plane and an image plane are placed parallel to each other at focal distance /. Collimated light is incident at the lens plane from the normal direction.
• a phase modulator (or lens) in the lens plane distorts the phase of the light, resulting in a curved phase function p( ), which corresponds to a local deflection of the light rays.
• a variable mirror is provided in the lens plane.
• phase delays introduced by a smooth phase function can be related to an equivalent, physical refractive lens under the paraxial approximation, which can be derived using either geometric optics or from the Hyugens principle.
• the paraxial approximation holds when sin ⁇ « ⁇ .
• the error in the paraxial approximation is less than 1 % . This facilitates optimizing directly for the phase surface.
• magnification factor m(. ) can be expressed in terms of the derivatives of the mapping between the lens and image planes (also see Figure 2):
• magnification, m, and therefore the intensity i u) on the image plane can be directly computed from the Laplacian of the scalar phase function on the lens plane.
• p( ) argmiri p ⁇ I — 1 + f ⁇ V 2 p( )) 2 d
• i p is a warped image
• i v (x) i(x + f ⁇ Vp( ))
• the target intensity i(u) in the image plane has been warped backwards onto the lens plane using the distortion u(x) produced by a given phase function p( ).
• phase update corresponds to solving a linear least squares problem with a discrete Laplace operator as the system matrix.
• CG Conjugate Gradient
• BICGSTAB BICGSTAB
• QMR Quasi Minimal Residual
• GPU graphics processor unit
• FIG. 4 shows algorithm stages for six iterations.
• the target image i gets progressively distorted through backwards warping onto the lens plane i ⁇ k) as the phase function p (k) converges towards a solution.
• the algorithm uses the undistorted target image to optimize an initial phase function.
• we update the target image on the lens plane by backward warping the image-plane target. This process increasingly distorts the target image for the modulator plane as the phase function converges.
• the backward warping step implies a non-convex objective function, we empirically find that we achieve convergence in only a small number of iterations (5-10). Overall processing time can be further accelerated by processing lower image resolutions first and upsampling the result.
• Convergence speed of this algorithm can be further improved by understanding that the computational cost of the method is due primarily to the solution of large-scale biharmonic problems.
• a Krylov subspace method QMR
• convergence is typically slow due to difficulties in finding an effective preconditioner and the scale of the systems.
• Algorithms useful for efficient solution of biharmonic systems are an ongoing topic of research, including, for example,
• proximal operator prox yP
• proximal operators contain a strictly convex regularization term, the whole operator is a strictly convex function even if F is only weakly convex. This property of proximal operators helps in designing algorithms with rapid convergence.
• Equation 7 By denoting the forward and inverse Fourier transforms as F & F _ 1 () respectively, complex conjugation by * and performing multiplication and division point- wise, the proximal operator for Equation 7 can be re-expressed in the Fourier domain as Equation 10 for Toeplitz matrices A.
• Equation 11 The constant a ⁇ 0 has been added to regularize the solver by favoring solutions with low curvature. This corresponds to solving a modified form of Equation 7 that imposes a penalty of - 1
• , as shown in Equation 11 p(x)
• V (1 1 1) argmin p(x) f (i p (x) - 1 + / ⁇ V 2 p(x)) 2 dx + J " (V 2 p(x)) 2 dx,
• Equation 11 is a non-linear variant of a common proximal algorithm, the proximal-point method, which is a fixed-point algorithm for minimizing an arbitrary convex F consisting of recursively calling prox yP by evaluating: p k+1 ⁇ - prox YF (p k ).
• the re-formulation of the algorithm results in orders of magnitude speedup to the algorithm when executed on a CPU using FFT based solvers over the QMR solver described above. If the per- frame computation times for a QMR solver are 20 minutes or more the Fourier version in Algorithm 1 may take approximately 0.6 seconds at the same resolution (256 x 128) on a Core i5 desktop computer, a speedup of approximately 2000 times.
• the conversion to Fourier domain solves also results in operations that are more easily implemented to run in parallel on one or more GPUs.
• the CUDA & CUFFT version of the code yields nearly a 150 times speedup over the single-threaded CPU version when run on a GeForce 770 GPU, resulting in roughly a 300,000 fold speedup over the naive CPU version implemented using QMR.
• the algorithm described herein is the first freeform lensing method of which the inventors are aware that is capable of operating in real-time, see Table 1. This is in contrast to methods such as [Schwartzburg et al. 2014], which produce satisfactory results, but have runtimes roughly five orders of magnitude higher than our GPU algorithm. This currently prevents their use in real-time capable projection systems.
• the algorithm is very well suited to hardware implementation on devices such as GPUs, FPGAs or ASICs due to its use of highly parallel FFTs and component- wise operations.
• Algorithm 1 for a fixed number of iterations (typically 10).
• Blender+LuxRender Although these models may not satisfy the paraxial approximation, they serve well for quick qualitative comparisons since thickness effects tend to manifest as low-spatial frequency distortions. Examples are shown in Figure 12 and 13 which illustrate the effect of mirror padding and the choice of a respectively. It is important to note that these distortions do not affect the prototype projector results since the prototype meets the conditions of the paraxial approximation well.
• phase modulation performs largely as expected, and the primary limitations in image quality are diffraction artifacts and speckle.
• phase function p( ) can be used directly to drive a digital phase modulation display (see below). However, if instead, we would like to create a refractive lens surface out of a transparent material, then this phase function may be converted to a geometric model for the lens shape.
• the lens material has a refractive index of n.
• Equations 12 and 13 as well as 9 t « dh(x)/dx, we can derive the lens shape as where h 0 is a base thickness for the lens.
• the height h(x) is a linear function of the phase.
• the refractive index n shows up only as a scalar multiplier to the phase function p(. ). Since p itself is approximately linear in the focus distance /, we can see that uniform scaling of the height field and uniform changes of the refractive index simply manifest themselves as a refocusing of the lens. This also shows that it is equivalently possible to adjust the example optimization procedure proposed above to directly optimize for h(. ) instead of p(. ).
• the formulation above may be preferable in cases where one is seeking to control only because a spatial phase modulator for example for applications in video projectors.
• Figure 5 and the right-hand image of Figure 5A show some example 3D printed refractive lenses.
• the left image shows the lenses themselves, while the center and right images show the caustics generated by them (the Lena image and a Siggraph logo). Due to resolution limits on the 3D printer, the lens dimensions have been optimized for large feature scales, which results in a short focal length.
• Figure 5 and the right-hand image of Figure 5A show results for goal-based caustics using refractive freeform lenses generated with our method.
• the lenses (shown on the left of Figure 5) were 3D printed on an Objet Connex 260 rapid prototyping machine using VeroClearTM material. Afterwards, the lenses were thoroughly cleaned and the fiat side was manually polished using fine grained sand paper and polishing paste.
• This type of 3D printer has a layer thickness of 42 ⁇ , which limits the feature size that can be readily created.
• the model can be rescaled to achieve different focal distances.
• focal distances f about 1" for the Siggraph logo and 5" for the Lena image.
• MEMS microelectromechanical systems
• Such devices include microelectromechanical systems (MEMS) based displays, such as the analog 2D array of mirrors fabricated by [Hoskinson et al. 2012], or deformable mirrors used in wavefront sensing and correction applications.
• Continuous deformable mirrors [Menn et al. 2007] seem a particularly attractive option since they eliminate diffraction due to regular pixel structures.
• the spatial resolution of these MEMS-based devices is still several order of magnitude lower than that of existing digital micro displays that are routinely used in digital projectors. This makes their use at this point, less attractive in a dual-modulation setup.
• Some embodiments advantageously apply wavefront modulators based on liquid crystal display (LCD) technology.
• LCDs are normally configured as amplitude (intensity) modulators by sandwiching them between two linear polarization filters. However, when operated without the second polarizer, they retard (modulate) the phase of passing light differently depending on the rotation state of the liquid crystals in each pixel. An electric field across the cell gap of each pixel controls the amount of phase retardation. In principle such a standard display is sufficient to implement a dynamic lens.
• micro displays that have been optimized to a) maximize the amount of phase retardation (on the order of 2 ⁇ and more) and to b) minimize the amount of polarization change.
• phase-only phase-only
• An example prototype embodiment used a reflective Liquid Crystal on Silicon (LCoS) chip distributed by [HOLOEYE ].
• This chip has a spatial resolution of 1920x1080 discrete pixels at a pixel pitch of 6.4 ⁇ , and can be updated at up to 60Hz. Access to a look-up-table allows for calibration of the modulator for different working wavelengths.
• the fill factor and reflectivity of the display are high compared to other technologies at 93% and 75% respectively.
• the phase retardation is calibrated to between 0 and 2 ⁇ , equivalent to one wavelength of light. This is sufficient to generate freeform lenses with a long focal distance. For shorter focal distances, we require more strongly curved wavefronts, which creates larger values for p(.) .
• We can address this issue by phase wrapping, i.e. just using the fractional part of p(.) to drive the SLM. This results in a pattern similar to a Fresnel lens.
• a first prototype contained a phase SLM without a second amplitude modulator, and is reconfigurable between two types of light source: a red 632.8nm HeNe laser, and a white LED. This prototype allows us to test the freeform lensing approach in isolation, and to evaluate artifacts such as diffraction based on light source type.
• a second prototype is a full dual-modulation projector using a green 532nm diode pumped solid state (DPSS) laser as a light source.
• DPSS diode pumped solid state
• the light path of a traditional projection system includes a high intensity light source and some form of beam shaping, for example beam expansion, collimation and homogenization, color separation and recombining optics.
• a small SLM At the heart of the projector, a small SLM attenuates the amplitude of light per pixel.
• Our prototype retained this architecture but replace the uniform illumination module with both a laser illumination and a phase SLM ( Figure 7).
• Our lensing system is inserted between the light source and the existing SLM, and forms an approximate light distribution on an intermediate image plane coinciding with the SLM plane.
• Figure 18 shows a selection of simulated and experimental results for our method.
• the first row of Figure 18 (“Phase Patterns”) shows the phase patterns computed by Algorithm 4.1 as applied to the phase modulator with black corresponding to no phase retardation and white corresponding to a retardation of 2 ⁇ . These patterns illustrate how phase patterns with maximum phase retardation larger than 2 ⁇ can be wrapped to the maximum phase retardation of the modulator, resulting in a pattern similar to a Fresnel lens.
• the second row of Figure 18 shows simulations of the phase pattern using the Huygens-Fresnel principle. Unlike geometric optics simulations such as path tracing, these simulations are able to capture many of the diffraction artifacts.
• the third row (“Direct") shows photos of our prototype using only phase modulation that exhibit diffraction artifacts as well as noise due to laser speckle. These artifacts can be almost entirely removed by introducing the diffuser in the fourth row of Figure 18 ("Diffused"); the photos for this row used identical camera settings to the "Direct" row.
• Phase Pattern the phase pattern as computed by Algorithm 1.
• Diffuser by adding the a thin-film diffuser, artifacts such as diffraction fringes nearly completely mitigated.
• Standard photo of standard, amplitude modulation only projection using a single amplitude modulator shows elevated black levels and low contrast.
• HDR Hexadjustment Reduction: Using our lensing approach redistributes light from dark regions to bright regions, resulting in improved black levels and increased highlight intensity. The last two rows appear slightly distorted due to an off-angle position of the camera which became necessary because of a short throw projection and close screen as well as baffles to block ambient light effectively to capture the black level of the system.
• Figure 5A shows the Lena image reproduced on the white light version of this setup.
• the broadband illumination averages out most of the diffraction artifacts, resulting only in a relatively small spatial blur, very similar to the backlight blur in the original dual modulation work by Seetzen et al. [2004] .
• This blur can be calibrated easily and can be compensated for in a dual modulation setup.
• Figure 9 shows just the effect of the freeform lensing approach, with the amplitude SLM set to a constant value.
• Figure 10 shows a result of the actual dual modulation approach.
• the second modulator stage has increase contrast and added significant detail, but cannot get rid of some of the high-frequency artifacts.
• HULLIN M. B., IHRKE, I., HEIDRICH, W., WEYRICH, T., DAMBERG, G., AND FUCHS, M. 2013 . State of the art in computational fabrication and display of material appearance. In Eurographies Annual Conference (STAR).
• Ng et al. 1999 Ng, M. K., Chan, R. H., and Tang, W.-C. 1999. A fast algorithm for deblurring models with neumann boundary conditions. SI AM Journal on Scientific Computing 21, 3, 851-866.
• NVIDIA NVIDIA
• O. PAPAS M., JAROSZ, W., JAKOB, W., RUSINKIEWICZ, S., MATUSIK,W., AND WEYRICH, T. 2011. Goal-based caustics.Computer Graphics Forum 30, 2, 503-511.
• SIGGRAPH SIGGRAPH
• a new, algorithm for freeform lens optimization (“goal-based caustics") that is dramatically simpler than some prior art algorithms.
• the algorithm may be applied to controlling the projection of light in real time or near real time.
• Some embodiments operate directly in phase space and therefore can be implemented as iterative methods that can not only generate modulation patterns for a phase modulator, but also for conventional refractive lenseswithout additional steps such as Poisson integration.
• a new dual-modulation projector design that combines one phase and one amplitude modulator for image generation and is capable of working white (incoherent) light.
• Methods and apparatus as described herein may also be applied for generating static light fields useful, for example, for architectural lighting and/or vehicle lighting.
• connection or coupling means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof;
• Embodiments of the invention may be implemented using specifically designed hardware, configurable hardware, programmable data processors configured by the provision of software (which may optionally comprise "firmware") capable of executing on the data processors, special purpose computers or data processors that are specifically programmed, configured, or constructed to perform one or more steps in a method as explained in detail herein and/or combinations of two or more of these.
• specifically designed hardware are: logic circuits, application-specific integrated circuits ("ASICs"), large scale integrated circuits (“LSIs”), very large scale integrated circuits (“VLSIs”), and the like.
• ASICs application-specific integrated circuits
• LSIs large scale integrated circuits
• VLSIs very large scale integrated circuits
• configurable hardware are: one or more
• programmable logic devices such as programmable array logic (“PALs”), programmable logic arrays (“PLAs”), and field programmable gate arrays (“FPGAs”)).
• PALs programmable array logic
• PLAs programmable logic arrays
• FPGAs field programmable gate arrays
• programmable data processors are: microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors, math co-processors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like.
• DSPs digital signal processors
• embedded processors embedded processors
• graphics processors graphics processors
• math co-processors general purpose computers
• server computers cloud computers
• mainframe computers mainframe computers
• computer workstations and the like.
• one or more data processors in a control circuit for a device may implement methods as described herein by executing software instructions in a program memory accessible to the processors.
• Processing may be centralized or distributed. Where processing is distributed, information including software and/or data may be kept centrally or distributed. Such information may be exchanged between different functional units by way of a communications network, such as a Local Area Network (LAN), Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channel.
• a communications network such as a Local Area Network (LAN), Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channel.
• processes or blocks are presented in a given order, alternative examples may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations.
• Each of these processes or blocks may be implemented in a variety of different ways.
• processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times.
• Software and other modules may reside on servers, workstations, personal computers, tablet computers, image data encoders, image data decoders, PDAs, color- grading tools, video projectors, audio-visual receivers, displays (such as televisions), digital cinema projectors, media players, and other devices suitable for the purposes described herein.
• aspects of the system can be practised with other communications, data processing, or computer system configurations, including: Internet appliances, hand-held devices (including personal digital assistants (PDAs)), wearable computers, all manner of cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics (e.g., video projectors, audio-visual receivers, displays, such as televisions, and the like), set-top boxes, network PCs, mini-computers, mainframe computers, and the like.
• PDAs personal digital assistants
• consumer electronics e.g., video projectors, audio-visual receivers, displays, such as televisions, and the like
• set-top boxes e.g., network PCs, mini-computers, mainframe computers, and the like.
• the invention may also be provided in the form of a program product.
• the program product may comprise any non-transitory medium which carries a set of computer-readable instructions which, when executed by a data processor, cause the data processor to execute a method of the invention.
• Program products according to the invention may be in any of a wide variety of forms.
• the program product may comprise, for example, non-transitory media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, or the like.
• the computer-readable signals on the program product may optionally be compressed or encrypted.
• the invention may be implemented in software.
• "software” includes any instructions executed on a processor, and may include (but is not limited to) firmware, resident software, microcode, and the like. Both processing hardware and software may be centralized or distributed (or a combination thereof), in whole or in part, as known to those skilled in the art. For example, software and other modules may be accessible via local memory, via a network, via a browser or other application in a distributed computing context, or via other means suitable for the purposes described above.
• image data is processed by a processor executing software instructions to yield control signals for a phase modulator.
• the software may execute in real time in some embodiments (other embodiments are also possible).

Landscapes

• Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Geometry (AREA)
• Liquid Crystal (AREA)
• Projection Apparatus (AREA)
• Optical Modulation, Optical Deflection, Nonlinear Optics, Optical Demodulation, Optical Logic Elements (AREA)
• Transforming Electric Information Into Light Information (AREA)
• Control Of Indicators Other Than Cathode Ray Tubes (AREA)
• Lenses (AREA)
• Liquid Crystal Display Device Control (AREA)

Priority Applications (12)

Applications Claiming Priority (4)

Related Child Applications (1)

Publications (1)

Family

ID=54765907

Family Applications (1)

Country Status (5)

Cited By (17)

Families Citing this family (36)

Citations (2)

Family Cites Families (36)

• 2015
  • 2015-06-03 EP EP15803468.6A patent/EP3152616B2/en active Active
  • 2015-06-03 CN CN202310048968.0A patent/CN116260951B/zh active Active
  • 2015-06-03 JP JP2016571109A patent/JP6715188B2/ja active Active
  • 2015-06-03 WO PCT/CA2015/050515 patent/WO2015184549A1/en not_active Ceased
  • 2015-06-03 CN CN201580037846.5A patent/CN106662739B/zh active Active
  • 2015-06-03 CN CN202010413872.6A patent/CN111641817B/zh active Active
• 2016
  • 2016-12-02 US US15/368,021 patent/US10003776B2/en active Active
• 2018
  • 2018-05-16 US US15/981,069 patent/US10404957B2/en active Active
• 2019
  • 2019-07-19 US US16/517,325 patent/US10834369B2/en active Active
• 2020
  • 2020-06-08 JP JP2020099433A patent/JP7104098B2/ja active Active
  • 2020-09-30 US US17/039,503 patent/US11558590B2/en active Active
• 2022
  • 2022-07-07 JP JP2022109538A patent/JP7511309B2/ja active Active
  • 2022-12-12 US US18/079,802 patent/US11877102B2/en active Active
• 2023
  • 2023-12-05 US US18/528,795 patent/US12574484B2/en active Active

Patent Citations (2)

Also Published As

Similar Documents

Legal Events