EP3631559A1 - Near-eye display with sparse sampling super-resolution - Google Patents
Near-eye display with sparse sampling super-resolutionInfo
- Publication number
- EP3631559A1 EP3631559A1 EP18708522.0A EP18708522A EP3631559A1 EP 3631559 A1 EP3631559 A1 EP 3631559A1 EP 18708522 A EP18708522 A EP 18708522A EP 3631559 A1 EP3631559 A1 EP 3631559A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- array
- elemental
- images
- display panel
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000005070 sampling Methods 0.000 title claims abstract description 26
- 238000009877 rendering Methods 0.000 claims abstract description 32
- 238000000034 method Methods 0.000 claims abstract description 21
- 239000002131 composite material Substances 0.000 description 10
- 238000010586 diagram Methods 0.000 description 10
- 230000008901 benefit Effects 0.000 description 8
- 230000004256 retinal image Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000002547 anomalous effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 239000011521 glass Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 210000001525 retina Anatomy 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004308 accommodation Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
- H04N13/307—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using fly-eye lenses, e.g. arrangements of circular lenses
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0118—Head-up displays characterised by optical features comprising devices for improving the contrast of the display / brillance control visibility
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B3/00—Simple or compound lenses
- G02B3/0006—Arrays
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B5/00—Optical elements other than lenses
- G02B5/005—Diaphragms
Definitions
- Head-mounted displays (HMDs) and other near-eye display systems can utilize an integral lightfield display or other computational display to provide effective display of three-dimensional (3D) graphics.
- the integral lightfield display employs one or more display panels and an array of lenslets, pinholes, or other optic features that overlie the one or more display panels.
- a rendering system renders an array of elemental images, with each elemental image representing an image or view of an object or scene from a corresponding perspective or virtual camera position.
- Such integral lightfield displays typically exhibit a tradeoff between resolution and eye relief (i.e., distance from which the user's eye can obtain a full field of view).
- Conventional near-eye display system employing an integral lightfield display typically sacrifice spatial resolution due to overlap in the lenslet projector-array.
- FIG. 1 is a diagram illustrating a near-eye display system employing pose detection and sparse sampling to provide increased display resolution in accordance with some embodiments.
- FIG. 2 is a diagram illustrating a conventional computational display in a near-eye display system.
- FIG. 3 is a diagram illustrating an example low fill-factor display for use in the near- eye display system of FIG. 1 in accordance with some embodiments.
- FIG. 4 is a diagram illustrating another example low fill-factor display for use in the near-eye display system of FIG. 1 in accordance with some embodiments.
- FIG. 5 is a diagram illustrating another example low fill-factor display for use in the near-eye display system of FIG. 1 in accordance with some embodiments.
- FIG. 6 is a flow diagram illustrating an example of a method of sparse sampling for rendering lightfield frames with increased resolution in the near-eye display system of FIG. 1 in accordance with some embodiments.
- FIG. 7 is a diagram illustrating an example sparse sampling operation for rendering lightfield frames in accordance with some embodiments.
- FIG. 8 is a diagram illustrating an example of light field super-resolution in
- FIGs. 1 -8 illustrate example methods and systems for sparse sampling super- resolution rendering of integral lightfield frames in a near-eye display system.
- the near-eye display system employs a computational display to display integral lightfield frames of imagery to a user so as to provide the user with an immersive virtual reality (VR) or augmented reality (AR) experience.
- Each integral lightfield frame is composed of an array of elemental images, with each elemental image representing a view of an object or scene from a different corresponding viewpoint.
- An array of lenslets overlies the display panel and operates to present the array of elemental images to the user as a single autostereoscopic image.
- near-eye display systems employing an integral lightfield display typically sacrifice spatial resolution due to overlap in the lenslet projector-array.
- the near-eye display systems described herein utilize sparse sampling of low fill-factor displays to recover some of the lost resolution by removing the overlap in image data within projector array elements.
- each of the projector array elements may be configured to sample a slightly different sub-region of a source image, and therefore have unique image data relative to one another.
- a sparse sampling of the source image can be obtained by the low fill-factor display to decrease the overlap of image data received at the lenslet projector-array and recover lost spatial resolution resulting from the decreased viewing distance of near-eye systems (e.g., less than 10 millimeters (mm) vs. the 10- 40 mm lens focal length of conventional light field displays vs 40+ mm for
- FIG. 1 illustrates a near-eye display system 100 incorporating sparse sampling super- resolution rendering of integral lightfield frames in accordance with at least one embodiment.
- the near-eye display system 100 includes a computational display sub-system 102 and a rendering component 104.
- the computational display sub-system 102 includes a left-eye display 1 10 and a right-eye display 1 12 mounted in an apparatus 1 14 (e.g., goggles, glasses, etc.) that places the displays 1 10, 1 12 in front of the left and right eyes, respectively, of the user.
- an apparatus 1 14 e.g., goggles, glasses, etc.
- each of the displays 1 10, 1 12 includes at least one display panel 1 18 to display a sequence or succession of integral lightfield frames
- each of the displays 1 10, 1 12 further includes an array 124 of lenslets 126 (also commonly referred to as "microlenses") overlying the display panel 1 18.
- the number of lenslets 126 in the lenslet array 124 is equal to the number of elemental images 122 in the array 120, but in other implementations the number of lenslets 126 may be fewer or greater than the number of elemental images 122. Note that while the example of FIG. 1 illustrates a 5x4 array of elemental images 122 and a
- a separate display panel 1 18 is implemented for each of the displays 1 10, 1 12, whereas in other embodiments the left-eye display 1 10 and the right-eye display 1 12 share a single display panel 1 18, with the left half of the display panel 1 18 used for the left-eye display 1 10 and the right half of the display panel 1 18 used for the right-eye display 1 12.
- Cross-view 128 of FIG. 1 depicts a cross-section view along line A-A of the lenslet array 124 overlying the display panel 1 18 such that the lenslet array 124 overlies the display surface 130 of the display panel 1 18 so as to be disposed between the display surface 130 and the corresponding eye 132 of the user.
- each lenslet 126 focuses a corresponding region of the display surface 130 onto the pupil 134 of the eye, with each such region at least partially overlapping with one or more adjacent regions.
- the rendering component 104 includes a set of one or more processors, such as the illustrated central processing unit (CPU) 136 and graphics processing units (GPUs) 138, 140 and one or more storage components, such as system memory 142, to store software programs or other executable instructions that are accessed and executed by the processors 136, 138, 140 so as to manipulate the one or more of the processors 136, 138, 140 to perform various tasks as described herein.
- processors such as the illustrated central processing unit (CPU) 136 and graphics processing units (GPUs) 138, 140
- storage components such as system memory 142
- software programs include, for example, rendering program 144 comprising executable instructions for a sparse sampling process, as described below.
- the rendering component 104 receives rendering information 148 from a local or remote content source 150, where the rendering information 148 represents graphics data, video data, or other data representative of an object or scene that is the subject of imagery to be rendered and displayed at the display sub-system 102.
- the CPU 136 uses the rendering information 148 to send drawing instructions to the GPUs 138, 140, which in turn utilize the drawing instructions to render, in parallel, a series of lightfield frames 151 for display at the left-eye display 1 10 and a series of lightfield frames 153 for display at the right- eye display 1 12 using any of a variety of well-known VR/AR computational/lightfield rendering processes.
- the CPU 136 may receive pose information 150 from an inertial management unit (IMU) 154, whereby the pose information 150 is representative of a current pose of the display sub-system 102 and control the rendering of one or more pairs of lightfield frames 151 , 153 to reflect the viewpoint of the object or scene from the current pose.
- IMU inertial management unit
- FIG. 2 illustrates a cross-section view 200 of a conventional computational display.
- Each of the lenslets 126 of the lenslet array 124 serves as a separate "projector” onto an eye of the user (e.g., eye 132 of FIG. 1 ), with each "projector” overlapping with one or more adjacent projectors in forming a composite virtual image 202 from the array 120 of elemental images 122 displayed at the display panel 1 18.
- the active refracting area (referred to herein as "fill-factor”) is nearly 100%. That is, the ratio of area that directs light towards the lenslet array 124 (i.e., the display panel 1 18) to the total contiguous area occupied by the occupied by the lenslet array including any gaps nears 100%.
- Displays with high fill-factors have overlapping data in multiple elemental images 122 from the virtual image 202.
- the lenslet 126-1 receives image data from elemental image 122-2 corresponding to region 206 of the virtual image 202.
- the lenslet 126-2 receives image data from elemental image 122-5 corresponding to region 206 of the virtual image 202 and the lenslet 126-3 receives image data from elemental image 122-8 corresponding to region 206 of the virtual image 202.
- the image data received from the elemental images 122-2, 122-5, and 122-8 have large amounts of overlapping information.
- Conventional displays having high fill-factors often overlap a number of elemental images (e.g., 4-6 elemental images) on the virtual image plane. This overlap results in the reduction of the source resolution by the same factor (i.e., reduction in resolution by 4x-6x).
- FIG. 3 illustrates a cross-section view 300 of a low fill-factor display that may be utilized in the near-eye display system 100 in accordance with some embodiments.
- Each of the lenslets 126 of the lenslet array 124 serves as a separate "projector" onto an eye of the user (e.g., eye 132 of FIG. 1 ), with each "projector” overlapping with one or more adjacent projectors in forming a composite virtual image 202 from the array 120 of elemental images 122 displayed at the display panel 1 18.
- the fill-factor is approximately 33%.
- the ratio of area that directs light towards the lenslet array 124 (i.e., the light emitting elements presenting elemental images 122 of display panel 1 18) to the total contiguous area occupied by the occupied by the lenslet array including any gaps is approximately 33%.
- Displays with lower fill-factors provide a sparse sampling of the source data, such as illustrated in FIG. 3, relative to high fill-factor displays.
- the lenslet 126-1 receives image data from elemental image 122-2 corresponding to region 206 of the virtual image 202.
- the lenslet 126-2 receives image data from elemental image 122-5 corresponding to region 206 of the virtual image 202 and the lenslet 126-3 receives image data from elemental image 122-8 corresponding to region 206 of the virtual image 202.
- the lower fill-factor display includes smaller light emitting elements that allow for more discrete attribution of source image data to reduce overlap of image data received at the lenslet array 124.
- the lenslet 126-1 receives image data from elemental image 122-2 corresponding to sub-region 206-1 within region 206 of the virtual image 202.
- the lenslet 126-2 receives image data from elemental image 122-5 corresponding to sub- region 206-2 within region 206 of the virtual image 202.
- the lenslet 126-3 receives image data from elemental image 122-8 corresponding to sub-region 206-3 within region 206 of the virtual image 202. Accordingly, while all of the lenslets 126-1 , 126- 2, and 126-3 take samples from the same local region of the source image (i.e., region 206), the lenslets 126 do not sample image data in the same exact location.
- Each of the lenslets 126-1 , 126-2, and 126-3 sample a slightly different sub-region of the source image, and therefore have unique intensity values (i.e., information content) of the image data relative to one another. Accordingly, a sparse sampling of the virtual image 202 can be obtained by the lower fill-factor display to decrease the overlap of image data received at the lenslets 126 and recover lost resolution by a ratio of 1 /(fill-factor). For example, the approximately 33% fill-factor display of FIG. 3 increases resolution of the virtual image 202 by approximately 3x (i.e., 1/0.33 fill- factor).
- the display panel 1 18 can include light emitting elements that are fabricated with a small fill-factor, such as illustrated in FIG. 3.
- Such light emitting elements can include, for example, organic light emitting diode (OLED) emitters.
- OLED organic light emitting diode
- Such light emitting elements are can be limited on their brightness output for a given current density per unit area and fill-factor ratio. In other words, sparse sampling by decreasing the fill-factor to increase virtual image resolution can sometimes have the consequence of lowering the brightness of the display panel 1 18. This lower brightness is due at least in part on smaller amounts of surface area available for light emitting element output.
- FIG. 4 illustrates a configuration for a low fill-factor display that may be utilized in the near-eye display system 100 in accordance with some embodiments.
- each of the lenslets 126 of the lenslet array 124 serves as a separate "projector” onto an eye of the user (e.g., eye 132 of FIG. 1 ), with each "projector” overlapping with one or more adjacent projectors in forming a composite virtual image 202 from the array of elemental images displayed at the display panel 1 18 by light emitting elements 402.
- the active refracting area (referred to herein as "fill-factor) is nearly 100%. That is, the ratio of area that directs light towards the lenslet array 124 (i.e., the display panel 1 18) to the total contiguous area occupied by the occupied by the lenslet array including any gaps nears 100%.
- a pixel aperture mask 404 is positioned between the light emitters of the display panel 1 18 and the lenslet array 124. It should be noted that the pixel aperture mask 404 does not actually reduce the fill-factor of the display panel 1 18. The surface area that directs light towards the lenslet array 124 is approximately 100%, similar to the configuration of FIG. 2. However, the additional pixel aperture mask 404 narrows light received at the lenslet array 124 and allows image data to be attributed to only a portion of the area of the light emitting elements 402.
- the light received at lenslet 126-1 past the pixel aperture mask may be attributed to a sub-region 408-1 of light emitting element 402-1 .
- the sub-region 408-1 of light emitting element 402-1 is analogous to FIG. 3's elemental image 122-2 and corresponds to sub-region 206-1 within region 206 of the virtual image 202.
- the aperture 410 may be attributed to a sub-region 408-2 of of light emitting element 402-2 and light received at lenslet 126-3 past the aperture 412 may be attributed to a sub-region 408-3 of light emitting element 402-3.
- These sub- regions 408-2 and 408-3 are analogous to FIG.
- the pixel aperture mask 404 allows for greater resolution by reducing the "effective fill-factor" (i.e., fill-factor as experienced from the point of view of the lenslets 126) without changing the actual fill-factor of the display panel 1 18, thereby increasing spatial resolution of the virtual image 202 while maintaining a larger emitter surface area (e.g., relative to FIG. 3) for increasing output brightness.
- the cross-section view 400 does not provide any details regarding the size and shape of the apertures in the pixel aperture mask 404.
- apertures may be used without departing from the scope of the present disclosure.
- substantially square or rectangular apertures may be used.
- cross-section view 400 illustrates a space between the pixel aperture mask 404 and the display panel 1 18 and the lenslet array 124, the spaces are provided for clarity and relational purposes. For example, in various
- the pixel aperture mask 404 may be disposed to be in direct contact with the display panel 1 18 or the lenslet array 124, or applied to a glass surface that is in direct contact with the display panel 1 18 or the lenslet array 124 without departing from the scope of the present disclosure.
- FIG. 5 illustrates another configuration for a low fill-factor display that may be utilized in the near-eye display system 100 in accordance with some embodiments.
- each of the lenslets 126 of the lenslet array 124 serves as a separate "projector” onto an eye of the user (e.g., eye 132 of FIG. 1 ), with each "projector” overlapping with one or more adjacent projectors in forming a composite virtual image 202 from the array of elemental images displayed at the display panel 1 18 by light emitting elements 402.
- the active refracting area (referred to herein as "fill-factor") is nearly 100%. That is, the ratio of area that directs light towards the lenslet array 124 (i.e., the display panel 1 18) to the total contiguous area occupied by the occupied by the lenslet array including any gaps nears 100%.
- a pixel aperture mask 404 is positioned between the light emitters of the display panel 1 18 and the lenslet array 124. It should be noted that the pixel aperture mask 404 does not actually reduce the fill-factor of the display panel 1 18. The surface area that directs light towards the lenslet array 124 is approximately 100%, similar to the configuration of FIG. 2. However, the additional pixel aperture mask 404 narrows light received at the lenslet array 124 and allows image data to be attributed to only a portion of the area of the light emitting elements 402.
- the light received at lenslet 126-1 past the pixel aperture mask may be attributed to a sub-region 408-1 of light emitting element 402-1 .
- the sub-region 408-1 of light emitting element 402-1 is analogous to FIG. 3's elemental image 122-2 and corresponds to sub-region 206-1 within region 206 of the virtual image 202.
- the aperture 410 may be attributed to a sub-region 408-2 of of light emitting element 402-2 and light received at lenslet 126-3 past the aperture 412 may be attributed to a sub-region 408-3 of light emitting element 402-3.
- These sub- regions 408-2 and 408-3 are analogous to FIG.
- the pixel aperture mask 404 allows for greater resolution by reducing the "effective fill-factor" (i.e., fill-factor as experienced from the point of view of the lenslets 126) without changing the actual fill-factor of the display panel 1 18, thereby increasing spatial resolution of the virtual image 202 while maintaining a larger emitter surface area (e.g., relative to FIG. 3) for increasing output brightness.
- the micro-lenses 502 are disposed between the display panel 1 18 and the pixel aperture mask 404, such that the micro-lenses 502 are configured to focus light emitted by the light emitting elements 402 past the apertures. Accordingly, combining the micro-lenses 502 with the pixel aperture mask 404 allows for greater resolution by reducing the "effective fill-factor" (i.e., fill-factor as experienced from the point of view of the lenslets 126) without changing the actual fill-factor of the display panel 1 18, thereby increasing spatial resolution of the virtual image 202 while using the micro-lenses 502 to focus light emitted from the larger emitter surface area (e.g., relative to FIG. 3) for increasing output brightness and light throughput to the lenslets 126.
- the "effective fill-factor" i.e., fill-factor as experienced from the point of view of the lenslets 1266
- the micro-lenses 502 to focus light emitted from the larger emitter surface area (e.g., relative to FIG. 3)
- cross-section view 500 does not provide any details regarding the size and shape of the apertures in the pixel aperture mask 404. Those skilled in the art will recognize that various sizes and shapes of apertures may be used without departing from the scope of the present disclosure. For example, substantially square or rectangular apertures may be used. It should further be noted that although the cross-section view 500 illustrates a space between the pixel aperture mask 404 and the display panel 1 18 and the lenslet array 124, the spaces are provided for clarity and relational purposes. For example, in various
- the pixel aperture mask 404 may be disposed to be in direct contact with the display panel 1 18 or the lenslet array 124, or applied to a glass surface that is in direct contact with the display panel 1 18 or the lenslet array 124 without departing from the scope of the present disclosure.
- FIG. 6 illustrates a method 600 of operation of the near-eye display system 100 for rendering lightfield frames using low fill-factor display panels to generate high resolution lightfield displays in accordance with some embodiments.
- the method 600 illustrates one iteration of the process for rendering and displaying a lightfield frame for one of the left-eye display 1 10 or right-eye display 1 12, and thus the illustrated process is repeatedly performed in parallel for each of the displays 1 10, 1 12 to generate and display a different stream or sequence of lightfield frames for each eye at different points in time, and thus provide a 3D, autostereoscopic VR or AR experience to the user.
- method 600 starts at block 602, whereby the rendering component 104 identifies the image content to be displayed to the corresponding eye of the user as a lightfield frame.
- the rendering component 104 receives the IMU information 152 representing data from various pose-related sensors, such as a gyroscope, accelerometer,
- the CPU 136 executing the rendering program 144, can determine a current pose of the apparatus 1 14 (e.g., HMD) used to mount the displays 1 10, 1 12 near the user's eyes. From this IMU information 152, the CPU 136, executing the rendering program 144, can determine a current pose of the apparatus 1 14 (e.g., HMD) used to mount the displays 1 10, 1 12 near the user's eyes. From this IMU information 152, the CPU 136, executing the rendering program 144, can determine a current pose of the apparatus 1 14 (e.g., HMD) used to mount the displays 1 10, 1 12 near the user's eyes. From this IMU information 152, the CPU 136, executing the rendering program 144, can determine a current pose of the apparatus 1 14 (e.g., HMD) used to mount the displays 1 10, 1 12 near the user's eyes. From this IMU information 152, the CPU 136, executing the rendering program 144, can determine a current pose of the apparatus 1 14 (e.
- the rendering program 144 manipulates the CPU 136 to sparsely sample the source object (e.g., virtual image 202 of FIG. 2) in the spatial domain and generate elemental images based on the determination in block 602 of imagery to be rendered.
- each elemental image represents an image or view of an object or scene from a corresponding perspective or virtual camera position, such as determined in block 602.
- sparsely sampling the source object includes minimizing the amount of image data within each generated elemental image that overlaps with image data within other elemental images at the virtual image plane.
- FIG. 7 illustrated is a diagram of a cross-section view 700 of a low fill-factor display (e.g., embodiment of FIG. 3) configured to sparsely sample the virtual image for display.
- each of the lenslets 126 of the lenslet array 124 serves as a separate "projector" onto an eye of the user (e.g., eye 132 of FIG. 1 ), with each "projector” overlapping with one or more adjacent projectors in forming a composite virtual image 202 from elemental images displayed at the display panel 1 18.
- the fill-factor is approximately 33%.
- the ratio of area that directs light towards the lenslet array 124 i.e., the light emitting elements, such as pixels 122, of the display panel 1 18
- the total contiguous area occupied by the occupied by the lenslet array including any gaps is approximately 33%.
- the pixels 122 represented by a given pattern are samples from the same local region (e.g., one of local regions 204-206) of the source virtual image 202.
- the pixels 122 are not sampled in exactly the same location, and therefore have unique intensity values (i.e. information content).
- the pixel 122-1 corresponds to a sparse sampling of sub-region 204-1
- pixel 122-4 corresponds to a sparse sampling of sub-region 204-2.
- both pixels 122-1 and 122-4 both sample in the same local region 204, they capture different perspectives and have unique intensity values relative to each other.
- pixel 122-2 corresponds to a sparse sampling of sub-region 206-1
- pixel 122-3 corresponds to a sparse sampling of sub-region 208-1
- an elemental image rendered based on the pixels 122-1 , 122-2, and 122-3 for presentation via lenslet 126-1 will contain unique data relative to, for example, an elemental image rendered based on pixels 122-4, 122-5, and 122-6 for presentation via lenslet 126-2 (as opposed to, for example, the display of FIG. 2 in which all three displayed lenslets receives more than one copy of the same virtual image data at the various lenslets).
- FIGS. 4-5 and their associated disclosure also discuss the generation of elemental images with greater spatial resolution and less overlap in image data between elemental images.
- the rendering program 144 manipulates the CPU 136 to sparsely sample the source object (e.g., virtual image 202 of FIG. 2) in the spatial domain and generate elemental images having no overlap in image data at the virtual image plane, thereby maximizing the efficiency of the system.
- the GPU subsequently renders the lightfield frame at block 606 and provides the lightfield frame to the corresponding one of the computational displays 1 10, 1 12 for display to the eye 132 of the user.
- the sparse sampling rendering operations described herein may be combined with varifocal lenses to shift the virtual image plane to achieve additional improvements to resolution of displayed imagery.
- FIG. 8 is a diagram illustrating an example of light field super-resolution in
- the image perceived at, for example, the retina of eye 132 of FIG. 1 of the user is the composite of a plurality of elemental images.
- Each of the elemental images passes through a lenslet (e.g., lenslet 126 of FIG. 1 ) within an array (e.g., array 124 of FIG. 1 ) and overlap to form the overlapping, composite image.
- retinal images of elemental images 802, 804, and 806 overlap to form an aggregate lightfield pixel grid 808 with overlapping subpixel elements (e.g., red subpixels 810, blue subpixels 812, and green subpixels 814).
- the lenslet array 124 is aligned with the pixel lattice grid, the retinal images of the subpixel elements overlap as shown.
- rotation of the lenslet array 124 relative to the display panel 1 18 results in light field super-resolution.
- the display pixels i.e., the sample grid
- the virtual image of the subpixels from each of the neighboring elemental images will no longer perfectly overlap.
- a rotation is a compound translation (e.g., x- and y-axis translation of subpixel position), thereby causing a deviation in the virtual images perceived at the retina.
- the rotation is modeled as a subpixel shift in the x- and y-axis positions of one elemental image's pixel grid relative to its neighbors' grids.
- the subpixels will be perfectly interleaved.
- the pixel grids of elemental images 816, 818, and 820 become separated by a half pixel in one direction and an integer number of half pixels in the other direction, resulting in the interlacing of adjacent elemental image's pixels.
- Image information is thus steered from adjacent elemental images into normally black areas of the composite retinal image 822 (e.g., due to non-emissive portions of the display between subpixels), thereby creating an increase of 1/N times the resolution, where N represents the number of elemental images sharing same pixel (e.g., the redundancy factor).
- N represents the number of elemental images sharing same pixel (e.g., the redundancy factor).
- the resulting composition retinal image 822 has a resolution which is increased by a factor of two.
- the additionally perceived pixels are contributions from adjacent elemental images that provide additional image information.
- subpixel 824 is the result of overlapping a blue subpixel and a green subpixel from two different elemental images.
- super-resolution by interleaving of subpixels may also be achieved by shifting lenslet-display distance (e.g. , z-axis distance between the lenslet array 124 relative to the display panel of FIG. 3) instead of the x- and y-axis translation due to lenslet-display rotation.
- the diopter distance between aliased overlap conditions i.e.
- the depth range of the virtual image (e.g. , virtual image 202 of FIG. 3) in such interleaved conditions is limited by lenslet depth-of-field or the user's eye accommodation range, such as to block perception of anomalous image
- certain aspects of the techniques described above may implemented by one or more processors of a processing system executing software.
- the software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium.
- the software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above.
- the non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like.
- the executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.
- a computer readable storage medium may include any storage medium, or combination of storage media, accessible by a computer system during use to provide instructions and/or data to the computer system.
- Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc , magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media.
- optical media e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc
- magnetic media e.g., floppy disc , magnetic tape, or magnetic hard drive
- volatile memory e.g., random access memory (RAM) or cache
- non-volatile memory e.g., read-only memory (ROM) or Flash memory
- MEMS microelect
- the computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).
- NAS network accessible storage
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762511561P | 2017-05-26 | 2017-05-26 | |
PCT/US2018/018434 WO2018217253A1 (en) | 2017-05-26 | 2018-02-15 | Near-eye display with sparse sampling super-resolution |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3631559A1 true EP3631559A1 (en) | 2020-04-08 |
Family
ID=68112757
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP18708522.0A Withdrawn EP3631559A1 (en) | 2017-05-26 | 2018-02-15 | Near-eye display with sparse sampling super-resolution |
Country Status (2)
Country | Link |
---|---|
EP (1) | EP3631559A1 (en) |
CN (1) | CN110325892A (en) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111338097A (en) * | 2020-04-18 | 2020-06-26 | 彭昊 | Spherical three-dimensional display |
CN111781736B (en) * | 2020-08-30 | 2023-07-14 | 成都航空职业技术学院 | 3D display device and method based on composite pinhole polaroid |
CN112859366B (en) * | 2021-04-01 | 2022-11-15 | 成都航空职业技术学院 | Double-vision 3D display method based on composite polaroid |
CN112859373B (en) * | 2021-04-01 | 2022-11-18 | 成都航空职业技术学院 | 3D display method based on discrete composite image element array |
CN113009709B (en) * | 2021-04-01 | 2022-11-15 | 成都航空职业技术学院 | Double-vision 3D display method based on composite pinhole array |
CN113031301B (en) * | 2021-04-01 | 2022-11-08 | 成都航空职业技术学院 | One-dimensional integrated imaging 3D display method with wide viewing angle and high optical efficiency |
CN113031297B (en) * | 2021-04-01 | 2022-11-15 | 成都航空职业技术学院 | Double-vision 3D display method based on polarized glasses |
WO2023156826A1 (en) * | 2022-02-18 | 2023-08-24 | Zhang Chuanrui | 2d transparent display device |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102012205164B4 (en) * | 2012-03-29 | 2021-09-09 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Projection display and method for projecting virtual images |
US20130285885A1 (en) * | 2012-04-25 | 2013-10-31 | Andreas G. Nowatzyk | Head-mounted light-field display |
US9880325B2 (en) * | 2013-08-14 | 2018-01-30 | Nvidia Corporation | Hybrid optics for near-eye displays |
CN109298526B (en) * | 2013-11-27 | 2021-09-07 | 奇跃公司 | Virtual and augmented reality systems and methods |
US9594247B2 (en) * | 2013-12-19 | 2017-03-14 | Nvidia Corporation | System, method, and computer program product for a pinlight see-through near-eye display |
US10437061B2 (en) * | 2015-08-03 | 2019-10-08 | Facebook Technologies, Llc | Near-ocular display based on hologram projection |
CN106097251B (en) * | 2016-06-22 | 2019-03-19 | 深圳信息职业技术学院 | Non-homogeneous sparse sampling video super-resolution method |
CN106019605B (en) * | 2016-08-04 | 2018-03-23 | 京东方科技集团股份有限公司 | Nearly eye display device and method |
-
2018
- 2018-02-15 CN CN201880008692.0A patent/CN110325892A/en active Pending
- 2018-02-15 EP EP18708522.0A patent/EP3631559A1/en not_active Withdrawn
Also Published As
Publication number | Publication date |
---|---|
CN110325892A (en) | 2019-10-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10764552B2 (en) | Near-eye display with sparse sampling super-resolution | |
EP3631559A1 (en) | Near-eye display with sparse sampling super-resolution | |
JP6886253B2 (en) | Rendering methods and equipment for multiple users | |
JP4538766B2 (en) | Imaging device, display device, and image processing device | |
JP4403162B2 (en) | Stereoscopic image display device and method for producing stereoscopic image | |
US9030737B2 (en) | 3D display device and method | |
CN1977544B (en) | 3D display method and apparatus | |
KR102185130B1 (en) | Multi view image display apparatus and contorl method thereof | |
EP3526967B1 (en) | Non-planar computational displays | |
US20170336626A1 (en) | Display device and display control method | |
WO2019105323A1 (en) | Display module, head-mounted display device, and stereoscopic image display method and apparatus | |
KR102121389B1 (en) | Glassless 3d display apparatus and contorl method thereof | |
KR20160021968A (en) | Method and apparatus for processing image | |
JP2004295013A (en) | Stereoscopic display device | |
JP2008085503A (en) | Three-dimensional image processing apparatus, method and program, and three-dimensional image display device | |
KR20120075829A (en) | Apparatus and method for rendering subpixel adaptively | |
KR20190131021A (en) | Near eye display with extended range | |
US20230023263A1 (en) | Multilens direct view near eye display | |
WO2019000948A1 (en) | Three-dimensional stereoscopic display panel, and display method and display apparatus therefor | |
US20170315348A1 (en) | Display device and display control method | |
KR101975246B1 (en) | Multi view image display apparatus and contorl method thereof | |
US9791707B2 (en) | Auto-stereoscopic display apparatus and non-transitory computer readable storage device | |
JP2004144874A (en) | Picture display device and picture display method | |
US10775617B2 (en) | Eye tracked lens for increased screen resolution | |
JP2010078883A (en) | Stereoscopic image display device and stereoscopic image display method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20190725 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN |
|
17Q | First examination report despatched |
Effective date: 20220218 |
|
18W | Application withdrawn |
Effective date: 20220309 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230519 |