WO2025015151A1

WO2025015151A1 - Perceptual-driven foveated optical see-through head-mounted displays

Info

Publication number: WO2025015151A1
Application number: PCT/US2024/037576
Authority: WO
Inventors: Hong Hua; Pengyinjie LYU
Original assignee: University of Arizona
Current assignee: University of Arizona
Priority date: 2023-07-13
Filing date: 2024-07-11
Publication date: 2025-01-16
Anticipated expiration: 2026-01-13

Abstract

Perceptual-driven foveated head-mounted display system, in which the angular pixel density decreases with an increase of field eccentricity according to a specified angular pixel density function. Minimally perceivable image artifacts and image resolution discontinuity can result optionally eliminating or limiting the need for an eyetracker or scanning mechanism. To simultaneously see a virtual image rendered by a display and a view of a real-world scene, a freeform prism is utilized as the optical combiner to merge the optical paths of the virtual image display and real-world view. Shapes of the optical surfaces of the freeform prism is controlled to yield spatially varying optical power to achieve a spatially varying angular pixel density distribution for a virtual display path. Exemplary designs yield a total field of view of over 80° while achieving a peak resolution of 2 arcminutes per pixel through a display panel of about 1900 by 1200 pixels.

Description

PERCEPTUAL-DRIVEN FOVEATED OPTICAL SEE-THROUGH HEAD- MOUNTED DISPLAYS Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63/526,574, filed on July 13, 2023, the entire contents of which application(s) are incorporated herein by reference. BACKGROUND Introduction Head-mounted displays (HMD) have been developed for many successful applications such as training, medical education, navigation, and others. Conventional HMD designs adopt the well-established rectilinear sampling method dominantly used for 2D display and imaging systems in which a finite number of pixels is spread evenly across an entire field of view. As a result, they are subject to the inherent trade-off between field of view (FOV) and resolution in the sense that for a given number of pixels the larger the FOV of an HMD is, the lower the angular resolution it has. For instance, consider an HMD design with an FOV of 120 degrees in its horizontal direction and a typical high-definition (HD) display device of 1920 x 1200 pixels. By spreading the 1920 pixels evenly in the horizontal direction, the angular resolution of a conventional HMD design system is about 3.75 arc minutes. Achieving an angular resolution of 1 arc minutes based on the same sampling method would require a display device of 7200x 4500 pixels and require data bandwidth 14 times that of a typical HD device. Such high-resolution displays are not only very challenging to produce, but also computationally challenging to process, transfer, and store such images. Besides these technical challenges, the rectilinear sampling scheme is very inefficient. It leads to large amount of redundant data for the human visual system to process because the visual acuity of the peripheral vision of the human eye drops off drastically beyond the fovea region of the retina. For instance, a 4K display based on a rectilinear sampling scheme only can support an HMD of 66 degrees circular FOV to achieve 1 arcminute angular solution matching with the resolution of 20/20 normal vision of human eyes, while about over 88% of the rendered information is not perceived by the human visual system at a given time instance. To mitigate the trade-off between FOV and resolution, a foveated display, inspired by the foveation properties of the human eye, can be generally characterized as a method that identifies a user’s region of interest (ROI) and allocates the limited resources, such as a finite number of pixels or data processing and transmission bandwidth, differently between the ROI and the peripheral area outside the ROI region. For example, the number of pixels allocated to a display region is a function of its distance to the center of the region of interest. The ROI may be determined by means of a gaze tracker, by tracking the salient points of scene rendered by the display, by pre-determining the ROIs of the scene, or by other mechanisms. A foveated display is often referred to as a gaze-contingent or eyetracked multi-resolution technique. Much effort has been made to apply foveation techniques in imaging and display applications. This prior research work on foveation techniques falls into one of three categories. The first category of work is experimental research on the perception and cognition to understand visual processing and perceptual artifacts produced by software simulated multi-resolution images. For instance, Loschki and McConkie investigated perceptual artifacts, such as perceptible image blur and image motion when viewing simulated foveated multi-resolution displays. The second category of work is an algorithmic approach in which foveation techniques are applied primarily to spatially variant image processing and video encoding and variable levels of detail graphics rendering to achieve real-time video communication and save data processing resources. In this approach, the display or imaging sensor hardware has a uniform high resolution, but the resolution of the rendered image is reduced away from the attended ROI. The third category of work takes a hardware approach, in which various imaging sensors or displays with spatially varying resolution are developed to reduce the requirements for high-resolution detectors and displays or high-quality and complex optical systems. For example, in the applications to imaging systems, Sandini et al described the implementation of a retina-like complimentary metal-oxide semiconductor (CMOS) sensor characterized by spatially variant resolution similar to that of the human retina and demonstrated that 35 times fewer pixels were needed in the spacevariant resolution sensor as compared with a constant high-resolution image of 1100 x 1100 pixels. Wick et al. presented the designs of foveated imaging systems in which a liquid crystal (LC) spatial light modulator (SLM) was used to dynamically correct the optical aberrations of a simple-structure wide FOV optics at the foveated region of interest (FRoI). Hua and Liu experimentally demonstrated a dual-sensor foveated imaging architecture where two separate imaging paths, one for foveal vision and one for peripheral vision, were integrated to capture foveated images; the high-resolution imaging path was steered by a 2D MEMS scanner according to the A-AOI, mimicking the eye movement of the human visual system (HVS). In application to display systems, Rolland presented the conceptual design of a head-mounted display with a high-resolution inset region where the high-resolution inset region can be optically relocated to different locations. More recently, Tan et al demonstrated a dual-resolution HMD design example with two display panels of different optical magnifications as the image sources and a switchable Pancharatnam-Berry phase deflector for shifting the position of the foveated view. A foveated display is a promising technique to realize a large FOV and high resolution at the same time in a display system design, yet only very few works exist in exploring a foveated display design, especially to foveated HMD designs. Among the limited prior work summarized above that attempted to apply a foveation method to the hardware design of an HMD system, a common method for implementing a foveated HMD is a dynamic discrete foveation approach where a foveated region offering a higher image resolution is dynamically steered in response to a user’s gaze direction and a relatively low-resolution region offers peripheral awareness. Such a dynamic foveation method typically utilizes a dual-display architecture in which two displays of different pixel resolutions or two different optical paths of different optical magnifications are utilized to render the foveated area and peripheral area with two different resolutions. The foveated area with high resolution typically covers a small (e.g. less than 10 degrees) FOV while the peripheral area covers a substantial portion of the entire FOV. The existing dual-display method also requires a mechanism to determine an instantaneous ROI toward which the display of higher resolution is aimed. In a dynamically foveated display, eye tracking is used to track the line of sight of a viewer and thus determine the intended ROI in real-time where a high-resolution image is rendered optically or digitally, while a low-resolution image is rendered for the background or a peripheral region. Finally, the existing dual-display architecture requires a scanning method to optically or mechanically steer and align the high-resolution foveated display approximately with the viewer’s line of sight to achieve the goal of foveated rendering. For example, Iwamoto et al demonstrated a bench prototype of a foveated display which used the eyetracked gaze point to control 2D opto-mechanical scanners to dynamically control the position of a high-resolution inset image over a wide FOV low-resolution background display. Rolland et al reported a high- resolution inset HMD system, in which a pair of microlens arrays optically duplicates a high- resolution inset image over a background display and a liquid crystal shutter is used to select one of the copies corresponding to the gazed ROI and block the others instead of using mechanically moving parts for scanning. In the more recent dual-resolution display system demonstrated by Tan et al, a switchable phase deflector was used to optically shift the position of the foveated image. Although a foveated HMD design based on such dual-display architecture made some progress as described above, this kind of foveated display is inevitably complex, high cost, large in volume, and heavy weight, because: multiple displays and imaging paths are necessary to render multi-level resolution displays; an eye tracking device is required to track ROI; and, a 2D steering mechanism, either mechanical or optical, is required for steering the foveated region. Finally, the multi-resolution approach provides multiple discrete samples of resolution and thus discontinuous perception of image quality as the eye moves, leading to visual artifacts. Perceptual-driven approach to foveated displays To overcome the drawbacks of the dynamically foveated method for HMDs, we recently proposed a new perceptual-driven static foveation approach to HMD designs. See also international publication WO2022031572 and US Patent application serial No. 18/018,549. Motivated by the fact that an HMD is generally attached to a user’s head with a relatively fixed viewing position, the characteristics of eye and head motions and the perceived visual effects are explored in the design process of a static foveation display, which offers a spatially varying angular resolution distribution through a single display path without requiring a gaze tracker and a 2D steering mechanism. Figure 3 shows the optical layout of our prior design of a static foveated HMD. (Pengyinjie Lyu and Hong Hua, “Design of a statically foveated display based on perceptual-driven approach,” Optics Express, 31(2): 2088-2101, 2023.) The design mainly comprises an eyepiece with spatially varying optical power and a microdisplay with regular pixel pitch. In this design, the microdisplay is an LCD display with a total of 2540 pixels in the diagonal direction, with a pixel pitch of 33µm. The eyepiece design comprises four optical elements. Three of the elements are off-the shelf optics available through Edmund Optics, while the last element is a custom-designed optics comprising two aspherical surfaces. The full FOV of the system is 80 degrees. Based on the design shown in Figure 3, Fig.4A illustrates the distortion grid of the foveated display at the display plane, Fig.4B plots the resolution distribution map across the virtual display plane, and Fig.4C further demonstrates the resolution distribution as a function of the field angle along the diagonal direction and the target resolution distribution is shown in dashed line for comparison. The rate of resolution degradation reached our design target. It remains nearly constant within the foveated region and drops from slow to fast speed as the field angle increases. In contrast, by utilizing microdisplays with a limited pixel resolution such as commonly available 2K pixels, a statically perceptual-driven foveated head-mounted display system of the present invention aims to offer a wide FOV across which the perceived image resolution appears high with imperceptible or minimal degradation and resolution discontinuity during eye movements. Compared to the multi-level discrete and dynamic foveation approaches before, the static foveation approach of the present invention will not only maintain resolution continuity, but also eliminate the need for an eyetracker or scanning mechanism and therefore minimize hardware complexity. SUMMARY Accordingly, in one of its aspects the present invention may provide a perceptual-driven foveated head-mounted display system for use in creating an image of a display panel that has a uniform pixel density. The system may include a wedge-shaped freeform prism having a spatially varying optical power as a function of a field angle wherein a virtual image of the display panel created by the wedge-shaped freeform prism has an angular pixel density that decreases with an increase of field eccentricity. The spatially varying optical power, ΦEP, of the wedge-shaped ௧^{^^( భ} m prism may be given by ^^_ா^( ^^) = _ల ⁾ freefor _{బ⋅ಷಷವ(ഇ)} , where is the pixel pitch of the display panel having a uniform pixel density, θ

a plane of the virtual image as seen by a viewer through the wedge-shaped freeform prism, and FFD is the angular pixel density in the virtual image. The angular pixel density, F_FD(θ), in the virtual image may be given by ^^_ி^( ^^) = ^ _{^^ୟ^ୟ୬ (^బ^^^మ(ఏ)ఃಶು(ఏ))}, where p0 is the pixel pitch of the display panel having a uniform pixel density, θ is the field angle of a pixel on a plane of the virtual image as seen by a viewer through the wedge-shaped freeform prism, and ^^_ா^ is the optical power of the wedge-shaped freeform prism. Based on a given optical design, the angular pixel density, FFD(θ), can be numerically _{obtained through raytracing, expressed as ^^} ^ℎ ^_{^ ^^( ^^) =} ^{^^( ^^+ ^^ ^^)−ℎ ^^( ^^)} ^_{^ ^^ , where the} ^h _c ^{( θ )} _{is the chief}

ray height at the display panel for a ^^ field, and ^^ ^^is the sampling interval of a densely sampled field grid. The perceptual-driven foveated head-mounted display system of the present invention may include a see-through auxiliary freeform prism disposed in optical communication with the wedge- shaped freeform prism at a surface thereof different from a surface of the wedge-shaped freeform prism proximate the display panel. The wedge-shaped freeform prism and see-through auxiliary freeform prism may cooperate to provide a real-world scene along a see-through path disposed through both the wedge-shaped freeform prism and see-through auxiliary freeform prism. A front auxiliary freeform prism may be provided in optical communication with the wedge-shaped freeform prism to provide a virtual display path therethrough extending from an exit pupil of the system to a first surface of the wedge-shaped freeform prism disposed proximate the display panel. In one configuration, the wedge-shaped freeform prism may be disposed (sandwiched) between the front auxiliary freeform prism and the see-through auxiliary freeform prism, where the front auxiliary freeform prism is disposed closest to the exit pupil. The front auxiliary freeform prism may include a first surface proximate the exit pupil, the first surface having an optical power to correct myopia or hyperopia of a user of the system, so that the front auxiliary freeform prism provides a vision correction lens. The perceptual-driven foveated head-mounted display system may include a first, second, and third optical surface at which light is refracted and/or reflected, the first optical surface configured to be disposed to receive light from the display panel, the second and third optical surfaces being relatively longer than the first optical surface, wherein at least one of the first, second, and third optical surfaces is a freeform optical surface and the first, second, and third optical surfaces are disposed relative to one another to provide a wedge-shape of the wedge-shaped freeform prism. The first, second, and third optical surfaces may all be freeform optical surfaces. One or more of the first, second, and third optical surface(s) may be a Zernike surface of the form

, Each surface may have a finite, non-zero radial distance, r, and at least one of the first, second, and third optical surface(s) may have one or more finite, non-zero Zernike polynomial coefficients, C_J+1, that provide the spatially varying optical power, Φ_EP, as a function of a field angle. The perceptual-driven foveated head-mounted display system may include a global coordinate system OXYZ wherein an origin of the global coordinate system OXYZ is positioned at the center of an exit pupil, a Z axis aligns with a head pose direction of a user, a OXY plane is perpendicular to the Z-axis, and a Y-axis is directing upward, and one or more of the first, second, and third optical surface(s) is symmetric about a YOZ plane perpendicular to an X-axis of the global coordinate system. In addition, the perceptual-driven foveated head-mounted display system may include a display panel disposed in optical communication with the wedge-shaped freeform prism. DRAWINGS The foregoing summary and the following detailed description of exemplary embodiments of the present invention may be further understood when read in conjunction with the appended drawings, in which: Figure 1A schematically illustrates a continuously statically foveated display scheme; Figure 1B illustrates an exemplary angular resolution distribution function in comparison to the visual acuity of a standard 20/20 human observer; Figure 2A illustrates a perceived maximum resolution and Figure 2B volume ratio of a foveation scheme as a function of eye rotation angles; Figure 2C illustrates data sampling efficiency as a function of the overall FOV for the foveation scheme; Figure 2C illustrates the total number of pixels required in a diagonal direction as a function of the overall FOV; Figure 3 schematically illustrates an exemplary configuration of an optical layout of a foveated head-mounted display of a prior design; Figure 4A illustrates a distortion grid of the foveated display of Fig.3; Figure 4B illustrates a resolution distribution in field map the foveated display of Fig.3; Figure 4C illustrates a resolution distribution and a target foveation scheme as the function of the field along the diagonal direction of the foveated display of Fig.3; Figure 5 schematically illustrates an exemplary configuration of a 2D optical layout of a perceptual-driven foveated head-mounted display system in accordance with the present invention; Figure 6 shows a resolution distribution map across the virtual display plane obtained by applying Eq. (7) and ray tracing; Figure 7 schematically illustrates a further exemplary configuration of a perceptual-driven foveated head-mounted display system in accordance with the present invention; Figure 8 schematically illustrates the YOZ plane layout of a virtual image optical path of the perceptual-driven foveated head-mounted display system of Fig.7; Figure 9 schematically illustrates the YOZ plane layout of an optical see-through path of the perceptual-driven foveated head-mounted display system of Fig.7; Figures 10A, 10B schematically illustrate an exemplary configuration of a layout of a reversed system with the Arizona eye model, with eye gaze at 0° and eye gaze at 30°, respectively; Figures 11A–11D illustrate MTF plots for various fields with rotation angle of the Arizona eye model, with Fig.11A at 0°, Fig.11B at 10°, Fig.11C at 20°, and Fig.11D at 30°; Figures 12A–12D illustrate resolution distributions of the perceptual-driven foveated head- mounted display system of Fig. 7 as the function of the field angle with the eye rotation angle, with Fig.12A at 0°, Fig. 12B at 10°, Fig.12C at 20°, and Fig. 12D at 30° along a diagonal direction; Figure 13A illustrates a distortion grid of the designed perceptual-driven foveated head- mounted display system of Fig.7; Figure 13B illustrates a resolution distribution of the designed perceptual-driven foveated head-mounted display system of Fig.7 as the function of the field along the diagonal direction; Figure 14A illustrates a MTF plot and Figure 14B a distortion grid of a perceptual-driven foveated head-mounted display system in accordance with the present invention; Figure 15 schematically illustrates an exemplary configuration of optical paths of upper and lower marginal rays of maximum and minimum Y-direction field; and Figure 16 illustrates a field sampling distribution by circular grid. DETAILED DESCRIPTION OF DISCLOSURE Overview of our perceptual-driven approach Referring now to the figures, wherein like elements are numbered alike throughout, Figure 1A shows a schematic illustration of a statically foveated display 100 where the display plane is the conjugate virtual image of a display source seen by a viewer through the optics in an HMD system; the optics are omitted for simplicity. Figure 1B plots an example of the resolution distribution for the display of Fig.1A as a function of field angles. Based on the characteristics of eye and head movement mechanisms of a user, a display plane 102 may be divided into three functional regions, a fovea region 102a, a parafovea region 102b, and a peripheral region 102c, Fig. 1A. The fovea region 102a is statically fixed at the center region of the display 100. It has uniform or nearly uniform pixel pitch offering the highest angular resolution. The fovea region is bounded by a critical field angle, ^^_^^, which is the visual and musculoskeletal balance point and defines a central region for frequent and comfortable eye movements. A preferred choice for ^^_^^ is between 5° and 20°, which is based on the physiological characteristics of eye movements reported by Burgess-Limerick et al. The parafovea region 102b of the display plane 102 is the annular zone immediately adjacent to the fovea region 102a. The virtual pixel pitch within the parafovea region 102b is expected to increase monotonically at a medium rate such that the angular resolution is still relatively high when the eye gaze falls within this region. The parafovea region 102b is bounded by two critical field angles, ^^_^^ and ^^_^ଶ, which are the balance points between eye movements and head motion. Due to muscular strain and discomfort, eye movements are expected to be gradually less preferred than head motion within the angular range of ±( ^^_^^, ^^_^ଶ) than the fovea region 102a. A preferred choice for ^^_^ଶ is between 20° and 40°, based on a report that head motion instead of eye movements likely occurs when the field angle is greater than 30°. The peripheral region 102c of the display plane 102 is immediately next to the parafoveal region 102b for field angles greater than ± ^^_^ଶ. The peripheral region 102c offers a rapid rate of degradation in resolution and mainly serves the purpose of peripheral vision and the sense of immersion. Within this region 102c, comfortable eye movements are unlikely to occur and head or body motion is preferred. In an HMD system, a head tracker is typically available for updating the scene rendering according to head motion and eye gaze much less likely falls within the peripheral region 102c. The resolution distribution function of a perceptual-driven foveated head-mounted display system in accordance with the present invention, FFD(θ), which is defined as the reciprocal of the angular resolution of the display in arcminutes, may be characterized by a three-segment function ^_{^^( ^^) when} ^| _^^ ^| _{≤ ^^^^} ^^_ி^( ^^) = ^ _{^^ଶ( ^^) when ^^^^ ≤} ^| _^^ ^| _{≤ ^^^ଶ} (1) ^^_ଷ( ^^) when | ^^| ≥ ^^_^ଶ _{where the pixel position P on the display plane 102 along the given direction} ^{^} r_{is specified by its} field angle, θ, which is defined as the angular deviation of the pixel position from the Z-axis of the OXYZ reference system in the unit of degrees and θ=0 corresponds to the direction matching with the Z-axis of the OXYZ reference coordinate system. _{θC 1} and _{θC 2} are the critical angles defining the boundaries of the regions, f₁, f₂, and f₃ are the segmented functions that characterize the relative resolution distribution within each corresponding region. To ensure resolution continuity and image smoothness near the center of the plane 102, which help to minimize transition artifacts across the center region, the function values at the first critical balance point, ^^_^^, should be equal. In other words, we may ensure that the following condition is satisfied: f ₁(θ )= f ₂ ( θ ) when θ = θ _{C 1} (2)

An example of the resolution distribution function in accordance with the present invention is given below: ^ ^{^} F 0 θ ^≤ θ C 1 ^_^ − ( θ − θ ) (3) where θ is the angular

at the display origin, σ a rate of resolution degradation within the parafovea region 102b, and ^^_௧ℎ is a threshold eye gaze angle that determines the eccentricity of a field angle in the peripheral region 102c and the rate of resolution degradation. ^^( ^^) is a polynomial correction function to make the third and the second segments of the resolution distribution function continuous at the ^^_^ଶ, where the specific order of the Δ (θ ) is determined by the value of σ, ^^_^^, and ^^_^ଶ. The form of the resolution distribution functions as well as the associated parameters in Eqs. (1) and (3) may be carefully optimized for minimally perceivable quality degradation or more data saving or ease of implementation. A set of performance metrics for perceived image quality and data sampling efficiency were detailed in our prior work by fully accounting for the characteristics of eye movement and head motion, target display specifications, and perceived performance as a function of eye motion and field angle. (P. Lyu and H. Hua, "Perceptual-driven approach to statically foveated head-mounted displays." Optics Express, 29(21), 33890-33914, (2021).) For instance, the metrics for perceived image quality include the perceived visual acuity, the perceived maximum resolution, and volume ratio. The perceived visual acuity of a foveated display, VA_FD, is used to quantify the perceived performance as a function of the field angle and eye gaze direction; the perceived maximum resolution characterizes the peak resolution across the overall FOV at different gaze directions; ands the volume ratio is a summative metric to evaluate the proportion of perceived resolution of the display below the perceptible limit of the HVS, which is calculated by the ratio of the integral volume enclosed by the perceived resolution curve of the display to the volume enclosed by the VA curve of the HVS across the display’s FOV for different eye gaze directions. The metrics for data sampling efficiency include the total raw data bandwidth, relative data sampling efficiency, and effective information throughput. The total raw data bandwidth is obtained by integrating its resolution distribution across all fields, the relative data sampling efficiency compares the sampling efficiency against a single-resolution uniformly sampled display which offers the same resolution across its entire FOV as the peak resolution of a foveated system, and the relative information throughput calculates the data bandwidth ratio between the HVS and a display scheme. Based on those metrics, the resolution distribution example in Fig.1B is expressed as: ^ F θ ^ p ^≤ 10 ^ eak 2 (4) where F_peak is the

angles are 10° and 30°, respectively. The foveation scheme in Eq. (4) was optimized for a foveated display offering a field angle of at least 80° and data sampling efficiency of at least 50%. The detailed optimization process has been described in our previous work. (P. Lyu and H. Hua, "Perceptual-driven approach to statically foveated head-mounted displays." Optics Express, 29(21), 33890-33914, (2021).) By setting the peak angular resolution to be 1 arcminute per pixel, Figs.2A and 2B show the perceived maximum resolution and volume ratio of the foveation scheme as a function of eye rotation angles varied from 0 up to 40° at an interval of 5°. The perceived maximum resolution is larger than 45% of peak resolution within the parafovea region 102b. The volume ratio in Fig.2B suggests that 95% or more of the entire display fields perform to the limit of the HVS for eye gaze angle up to ±25^◦, which indicates nearly no perceivable degradation within this range of eye motion. Figures 2C and 2D show the relative data sampling efficiency and the total number of required pixels in the diagonal direction as a function of the overall FOV, respectively. The foveation scheme thus offers a high data sampling efficiency of 92.8% for displays of 160° FOV, and such a statically foveated scheme can be adequately implemented with a 4K display covering over 160° full FOV. Initial design example To design a foveated system offering a spatially varying angular resolution distribution, one method is to utilize a display offering spatially varying pixel density that matches with the angular resolution distribution of a foveated scheme. Practically, however, it is challenging to manufacture displays with spatially varying pixel density. Instead, in accordance with the present invention, a perceptual-driven foveated head-mounted display system can be realized by adopting display panels with uniform pixel density and carefully designing an eyepiece offering spatially varying optical power _{Φ EP} (i.e., the reciprocal of the eyepiece focal length ^^_ா^), such that the angular resolution distribution of the virtual display seen through the eyepiece corresponds to a desired foveated scheme. System overview for a perceptual-driven foveated head-mounted display system Our prior work demonstrated the feasibility of applying a statically foveated scheme to an eyepiece design for an immersive HMD. This type of eyepiece layout as shown in Fig.3, however, is not feasible for a perceptual-driven foveated head-mounted display system. The design of a perceptual-driven foveated head-mounted display system in accordance with the present invention involves the choice of a proper optical combiner which merges the optical paths of virtual image display and real-world view. Though there are several choices of optical combiner techniques, from a simple planar beamsplitter to a sophisticated holographic waveguide combiner, few of these combiners support large FOV, compact dimensions, and high optical performance. In designs of the present invention, after considering various tradeoff factors, we chose to utilize a freeform waveguide combiner for a perceptual-driven foveated head-mounted display system. However, it is worth noting that a similar foveation approach and optimization strategy can be applied to designs based on other optical combiner techniques by those skilled in the art. Overall specifications are provided in Table 1 for an exemplary system of the present invention. To meet the requirements for a minimal interpupillary distance (IPD) of 55mm and desired optical performance, we opted for a 2-inch display with a resolution of 1920 x 1840 pixels and a pixel pitch of 20 μm. The active area utilized was 1920 x 1230 pixels, corresponding to a size of 36.5 mm x 23.5 mm. In order to accommodate the range of eye movement for eye pupils ranging from 2 to 4 mm without eye tracking device, the exit pupil diameter of the eyepiece was increased to 10 mm. The eye clearance was set to be greater than 15 mm, and the FOV target covered ±35° horizontally and from +25° to -15° vertically, resulting in an approximate diagonal FOV of 80 degrees. Table 1. The overall specifications of an exemplary system of the present invention. Parameters Specifications Display Active panel size 36.5 mm x 23.5 mm Active pixel resolution 1920 x 1840 pixels Pixel pitch 0.019 mm Exit pupil diameter 10 mm Eye relief >=15 mm Material of freeform prism COP (Cyclo Olefin Polymer) Optical system Wavelength 480-625 nm FOV 80° diagonally, ~70° (H) * ~40° (V) Vignetting No Image quality MTF > 10% at 27 cycle/mm Embodiment #1 of a perceptual-driven foveated head-mounted display system A first exemplary embodiment of the present invention adopts a monolithic freeform wedge prism structure as the main eyepiece attached to an auxiliary freeform prism, aka compensator, for the see-through optical path. The 2D layout of the eyepiece optics design for a perceptual-driven foveated head-mounted display system 500 in accordance with the present invention is shown in Fig. 5. The eyepiece optics may include a main freeform prism 502 and a freeform auxiliary freeform prism 504. The main freeform prism 502, when combined with a microdisplay panel 506, produces a magnified virtual image for the virtual display path. The freeform auxiliary freeform prism 504, combined with the main freeform prism 502, serves as a compensator to receive a real-world image for the see-through light path. In one exemplary configuration the main freeform prism 502 may include three optical surfaces, S₁-S₁’, S₂, and S₃. The surface S₂ may include a half-mirror coating to serve as the combining surface for the paths of virtual display and see-through view. The freeform auxiliary freeform prism 504 may include two optical surfaces S2’ and S4, where S2’ shares the same optical prescription as S₂. In the virtual display path, the light rays emitted from the display panel undergo a first refraction at surface S₃. Subsequently, two consecutive reflections occur at surfaces ^^^′ ^ and S₂, with a total internal reflection taking place at surface ^^^′ ^ and a reflection being caused by the half-mirror coating at surface S2. The reflected light rays then are refracted by surface S1 and are directed toward the exit pupil of the system. In the see-through path, the light rays from a real-world scene are propagated by a sequence of refractions through the surfaces of the auxiliary freeform prism 504 and main freeform prism 502. A main difference of this embodiment from the prior designs is that the three optical surfaces of the main freeform prism 502 are carefully optimized (optimization methods are detailed below) to produce spatially varying optical power, ΦEP(θ), which is the reciprocal of the prism focal length, expressed as ^^ _{^^ ^^}( ^^) = ¹ ^_{^ ^^ ^^( ^^)} (5) where, as illustrated in Fig.1, θ

plane 102 seen by a viewer through the eyepiece optics, which is optically conjugated to a microdisplay source, in the unit of degrees. The field angle, θ, may be decomposed into θ_x and θ_y, corresponding to the X-component and Y-component of the field angle θ in the horizontal and vertical directions, respectively. When combined with a microdisplay panel 506 with a uniform pixel density, the main freeform prism 502 produces a virtual display image with a spatially varying pixel density distribution to achieve a perceptual-driven foveated head-mounted display system. The spatially varying pixel density distribution of the virtual display can be characterized by a resolution distribution function, F_FD(θ), which is defined as the reciprocal of the angular resolution of the display in arcminutes, and can be calculated from the optical power distribution of the eyepiece as, ^^ _{^^ ^^}( ^^) = ¹ 6_{0atan ( ^^ ^^ ^^2( ^^) ^^ ^^ ^^( ^^))}, (6) where p is t

0 he pixel on the virtual display plane seen by a viewer through the eyepiece optics, and Φ_EP is the optical power of the main freeform prism 502. The process of optimizing the freeform prism 502 to obtain a spatially varying optical power is detailed below under the section heading “optimization method.” Once an optimized design is obtained, we can calculate the resolution distribution in optical design software by tracing the chief ray height intersections at the display panel 506 for a grid of densely sampled field angles. When the field sampling is sufficiently dense to ensure the difference between two adjacent fields is small enough, the resolution distribution at a field angle of θ can be characterized as: ^^ ^{ℎ ( ^^+ ^^ ^} ^_{^ ^^}( ^^) = ^{^^ ^)−ℎ ^^( ^^)} ^_^ (7) 0_{× ^^ ^^} where the ℎ_^( ^^) is the

interval of sampled field grids. For example, ^^ ^^ may be set to be about 0.1% of the full FOV or even smaller intervals. By plugging the Eq. (7) into Eq. (6), the field-dependent optical power of the eyepiece can be obtained as: ^^{^ ^^ ^^( 1} 6_{0⋅ ^^ ^^ ^^( ^^)} ⁾ ^^ _{^^ ^^}( ^^) = ^{^^ ^^ ^^2( ^^)} (8) Tables 2 and 3 provide

invention for a perceptual-driven foveated head-mounted display system design that satisfies the system specifications defined in Table 1. In this embodiment, each of the freeform optical surfaces, S₁-S₁’, S₂, S₃, and S₄ are defined using a global reference coordinate system OXYZ. As illustrated in Fig. 5, the origin of the global coordinate system OXYZ is positioned at the center of the exit pupil, where the eye pupil of a viewer is placed. The Z axis aligns with the head pose direction, the OXY plane is perpendicular to the Z-axis, and the Y-axis is directing upward. It is assumed that the display panel 506 is located above the eyebrow, thus the light ray path for the virtual display is folded within the main freeform prism 502 along the Y direction through two reflections by the surfaces S1’ and S2, respectively. In the design configuration, all optical surfaces are decentered and tilted relative to this global reference. In this embodiment, the specifications of surface S₂ of main freeform prism 502 are the same as those for S₂’ of the auxiliary freeform prism 504. Furthermore, all of the optical surfaces are specified with a Zernike polynomial with symmetry about the YOZ plane. A Zernike polynomial-based freeform surface can be defined as: (9)

The global coordinates and the specific surface parameters of the freeform surfaces forming the prism and the lens are listed in Table 2 and 3, respectively. Table 2 The global coordinates of the freeform surfaces forming main freeform prism 502 and auxiliary freeform prism 504 and display panel 506 Definition of the local surface references in the global coordinate system OXYZ. Origin of surface reference Orientation of the surface Rotation X (mm) Y (mm) Z (mm) about X-axis θ (°) Surface S1-S1’ 0 -6.922 37.369 30.990 Origin: O1 (x1, y1, z1) Orientation: θ₁ Surface S2-S2’ 0 5.515 69.027 -20.100 Origin: O2 (x2, y2, z2) Orientation: θ2 Surface S3 0 20.914 19.562 57.998 Origin: O₃ (x₃, y₃, z₃) Orientation: θ₃ Surface S4 0 85.601 84.940 59.754 Origin: O4 (x4, y4, z4) Orientation: θ4 Microdisplay 0 36.063 36.507 51.122 Origin: O_Im (x_Im, y_Im, z_Im) Orientation: θIm Table 3 The parameters of freeform optical surfaces defined with Zernike surface type Coefficients comment Surface ^^_^ Surface ^^_ଶ-S ’ Surface ^^_ଷ Surface ^^_ସ Y Radius R -415.48745 243.901518229346 -20 -61.9308447245938 C69 Normalization Radius 100 100 100 100

applying Eq. (7) and ray tracing. Embodiment #2 of a perceptual-driven foveated head-mounted display system We also demonstrate a further exemplary embodiment of freeform prism design for a perceptual-driven foveated head-mounted display system 700 in accordance with the present invention. To minimize see-through distortion and simplify the second auxiliary freeform prism design 704, we introduce another auxiliary freeform prism 705 between the wedge-shaped main freeform prism 702 and the human eye, creating a sandwich structure, Fig.7. The first auxiliary freeform lens 705 may include ^^1 and ^^2, the main freeform prism 702 may include ^^3, S4, ^^3, and ^^5, and the second auxiliary freeform prism 704 may include ^^4′ and ^^6. In this structure, the perceptual-driven foveated head-mounted display system 700 may include three elements: the main wedge prism 702, the first auxiliary freeform prism 705 attached to the front surface of the main freeform prism 702, and the second auxiliary freeform prism 704 attached to the back surface of the main freeform prism 702. The combination of auxiliary freeform prism 705 and main freeform prism 702 serves as the optical system for the virtual display path, while the combination of auxiliary freeform prism 705, main freeform prism 702, and second auxiliary freeform prism 704 serves as the optical system for the see-through light path. The main freeform prism 702 may include three optical surfaces, S3-S3’, S4, and S5. The surface S4 may be facilitated by a half-mirror coating and serves as the combining surface for the paths of virtual display and see-through view. The first auxiliary freeform prism 705 may include two optical surfaces, S1 and S2, where S2 preferably shares the same optical prescription as S3- S3’. The second auxiliary freeform prism 704 may include two optical surfaces, S4’ and S6, where S4’ preferably shares the same optical prescription as S4. In the virtual display path, the light rays emitted from the display panel 706 undergo a first refraction at surface ^^5. Subsequently, two consecutive reflections occur at surfaces ^^3^′ and S4, with a total internal reflection taking place at surface ^^3^′ and a reflection being caused by the half- mirror coating at surface S4. The reflected light rays then are refracted by surface ^^3 and are directed toward the first auxiliary freeform prism 705. Following two consecutive refractions by S2 and S1 of the prism 705, the light rays originated from the display panel finally reach the exit pupil, where a viewer’s eye is located for observing the virtual image. A small air gap is required between elements 705 and main freeform prism 702 to maintain the total internal reflection for the reflection on the S3’ in the display path. In the see-through path, the light rays from a real-world scene are propagated by a sequence of refractions through the surfaces of the second auxiliary freeform prism 704, main freeform prism 702, and 705. The first and second auxiliary prisms 705, 704, positioned on either side of the main freeform prism 702, form a plane parallel plate ensuring an undistorted see-through view of the real scene. This 3-element sandwich structure not only mitigates distortion of the see- through view but also offers potential capabilities for vision correction. For users with normal vision, the optical surfaces ^^1 and ^^6 can remain flat to ensure excellent see-through performance and minimal distortion. However, for users with myopia or hyperopia, the curvature of optical surface ^^1 can be adjusted to accommodate different diopter correction requirements. In such cases, the curvature of ^^1 can be modified to be negative or positive to address the specific needs of individual users. In the meantime, surface ^^6 can be optimized to minimize both shift and distortion of the real-world scene. In summary, the first auxiliary freeform prism 705 can serve as a vision correction insert lens, eliminating the need for additional eyeglasses. This is precisely why a 15-mm eye clearance has been selected for the design. A fundamental difference of this invention from the prior efforts lies in the fact that the optical system of the virtual display path, composed of auxiliary freeform prism 705 and main freeform prism 702, yields a desired spatially varying optical power as a function of the field angle, to achieve a perceptual-driven foveated head-mounted display scheme. Among the three elements in Fig.7, the main freeform prism 702 provides most of the optical magnification for the virtual display path, thus it is the core element to be optimized for achieving spatially varying optical power in our perceptual-driven foveated head-mounted display system scheme. The optical surfaces (S1 through S5) forming the two elements 705 and main freeform prism 702 are carefully optimized together (optimization methods are detailed below) to produce spatially varying optical power, Φ _EP , which is the reciprocal of the prism focal length f _{E P} , expressed by Eq. (5). When combined with a microdisplay panel 706 with a uniform pixel density, the combination of the elements 705 and main freeform prism 702 produces a virtual display image with a spatially varying pixel density distribution to achieve a perceptual-driven foveated head- mounted display system 700. The spatially varying pixel density distribution of the virtual display can be characterized by a resolution distribution function, FFD(θ), which is defined as the reciprocal of the angular resolution of the display in arcminutes, and can be calculated from the optical power distribution of the optical system by applying Eq. (6). The process of optimizing a combined optical system of auxiliary freeform prism lens 705 and main freeform prism 702 to obtain a spatially varying optical power is detailed below. Once an optimized design is obtained, we can calculate the resolution distribution in optical design software by tracing the chief ray heights intersected at the display panel 706 for a grid of densely sampled field angles. When the field sampling is sufficiently dense to ensure the difference between two adjacent fields is small enough, the resolution distribution at a field angle of θ can be characterized using Eq. (7). Based on the structure in Fig. 7, Figure 8 shows the 2D layout of the virtual optical path for an optimized embodiment of a perceptual-driven foveated head-mounted display system 700 that satisfies the system specifications defined in Table 1. Figure 8 provides a cross-sectional view of the YOZ plane showcasing the elements that are relevant to the virtual optical path, namely the first auxiliary freeform prism 705 and the main freeform prism 702. By incorporating the first auxiliary freeform prism 705, the optical see-through path can easily achieve exceptional performance with a flat surface ^^6 of the second auxiliary freeform lens 704, while the other surface ^^4^′ of the second auxiliary freeform prism 704 is picked up from the surface ^^4. The second auxiliary prism 704 exhibits a minimum thickness of approximately 1.5 millimeter. The layout of the optical see-through path for the perceptual-driven foveated head- mounted display system 700 is depicted in Fig.9. Tables 4 and 5 provide specifications of the embodiment shown in Figs. 7–9 for the perceptual-driven foveated head-mounted display system design 700. In this embodiment, each of the freeform optical surfaces, S1 through S6 are defined using a global reference coordinate system OXYZ. As illustrated in Fig.7, the origin of the global coordinate system OXYZ is positioned at the center of the exit pupil, where the eye pupil of a viewer is placed. The Z axis aligns with the head pose direction, the OXY plane is perpendicular to the Z-axis, and the Y-axis is directing upward. It is assumed that the display panel 706 is located above the eyebrow, thus the light ray path for the virtual display is folded within the main freeform prism 702 along the Y direction through two reflections by the surfaces S3’ and S4, respectively. In the design configuration, all optical surfaces are decentered and tilted relative to this global reference. Furthermore, surfaces S1 and S6 are optical flats and the remaining optical surfaces, including S2, S3-S3’, S4-S4’, and S5, are specified with a Zernike polynomial with symmetry about the YOZ plane. A Zernike polynomial-based freeform surface can be defined using Eq. (9). In this embodiment, the specifications of surface S2 of Element 1 are the same as those for S3-S3’ of Element 2, and the specifications of surface S4’ of Element 3 are the same as those surface S4 of Element 2. The global coordinates and the specific surface parameters of the freeform surfaces forming the prism are listed in Table 4 and 5, respectively. Table 4 The global coordinates of the freeform surfaces of the main freeform prism 702 and display panel 706 Definition of the local surface references in the global coordinate system OXYZ. Origin of surface reference Orientation of the surface Rotation X (mm) Y (mm) Z (mm) about X-axis θ (°) Surface S3-S3’ 0 -8.584 27.249 40.535 Origin: O3 (x3, y3, z3) Orientation: θ3 Surface S4 0 -26.361 68.442 -29.208 Origin: O₄ (x₄, y₄, z₄) Orientation: θ₄ Surface 5 0 22.410 -7.221 81.437 Origin: O5 (x5, y5, z5) Orientation: θ5 Microdisplay 0 32.06227 34.71714 77.471 Origin: O_Im (x_Im, y_Im, z_Im) Orientation: θIm Table 5 The parameters of the freeform surfaces with Zernike surface type Coefficients comment Surface ^^_ଶ, ^^_ଷ, & ^^_ଷ ^ᇱ Surface ^^_ସ, & ^^_ସ ^ᇱ Surface ^^_ହ Y Radius R -117.4370384731 -85.2090530247876 -115.234511667086 C69 Normalization 100

100 100 Radius C1 k -1.36515814490312 1.2171746688979 1.99346959613328 C2 1 (piston) 0 0 0 C3 Rcosθ 0 0 0 C4 Rsinθ 0 0 0 C5 R²cos(2θ) -16.5774862327263 39.8807446839406 37.2269452450524 C6 2R²-1 -11.5078328629417 69.6377858885028 13.0873317915867 C7 R²sin(2θ) 0 0 0 C8 R³cos(3θ) 0 0 0 C9 (3R³-2R)cosθ 0 0 0 C10 (3R³-2R)sinθ -29.5209693142432 9.37018625756946 52.571231092239 C11 R³sin(3θ) 64.4426067748386 -13.3880989186106 -82.3370466962103 C12 R⁴cos(4θ) -30.14846152253 29.8005694801582 77.2709362219925 C13 (4R⁴-3R²)cos(2θ) -4.55206191181011 7.07014379837453 18.9324289910254 C14 6R⁴-6 R²+1 -5.87157634342453 46.8556156661628 -21.3334192151897 C15 (4R⁴-3R²)sin(2θ) 0 0 0 C16 R⁴sin(4θ) 0 0 0 C17 R⁵cos(5θ) 0 0 0 C18 (5R⁵-4R³)cos(3θ) 0 0 0 C19 (10R⁵- 0 0 0 12R³+3R)cosθ C20 (10R⁵- -9.47229517570933 -0.20779991451017 -25.289876352918 12R³+3R)sinθ C21 (5R⁵-4R³)sin(3θ) 14.0539904483067 -2.01820056836617 -64.5917818221494 C22 R⁵sin(5θ) -28.4815624578547 -29.3314703571434 -92 C23 R⁶cos(6θ) 21.4339942207105 -34.2275921880354 92 C24 (6R⁶-5R⁴)cos(4θ) -8.42661690375713 6.24328406715904 40.6148950716777 C25 (15R⁶- 0.053736942498536 -2.05480336882794 35.3998896025369 20R⁴+6R²)cos(2θ) 7 C26 20R⁶-30R⁴+12R²- 0.030145919028660 10.7136736713839 -16.8373303370225 1 5 C27 (15R⁶- 0 0 0 20R⁴+6R²)sin(2θ) C28 (6R⁶-5R⁴)sin(4θ) 0 0 0 C29 R⁶sin(6θ) 0 0 0 The image performance of conventional HMD designs based on rectilinear sampling method is commonly evaluated by examining the modulation transfer functions (MTF) in the microdisplay space across the FOV for spatial frequencies up to Nyquist frequency set by the display pixel pitch. Due to the nature of the spatially varying optical power of the eyepiece in the foveated eyepiece design 700, it is no longer valid to adopt this common practice. Instead, to analyze the perceived performance of the foveated eyepiece design 700 in the visual space, we flipped the optical system left-to-right in Fig.8, traced rays from display 706 toward the exit pupil and inserted the Arizona eye model at the exit pupil. The reversed layout is illustrated in Figs.10A, 10B, where the display panel 706 is set to be the object with uniform spatial sampling and the retina of the eye model is treated as the image plane. The entrance pupil of the eye model, 3.05mm away from the cornea, is located at the exit pupil position of the foveated eyepiece. To simulate the effects of eye movement, the Arizona eye model is set to rotate around its rotation center, which is 13mm behind the corneal vertex. In Figures 10A and 10B, we present two examples of the reversed layouts obtained by rotating the Arizona eye model by 0° and 30° in the YOZ plane, respectively. The perceived performance of the perceptual-driven foveated head-mounted display system was assessed by rotating the Arizona eye model along the diagonal direction across the full FOV. The Arizona eye model was rotated from 0° to 30° with a 10° interval along the diagonal direction for full FOV, corresponding to the positions (0°, 0°), (8.14°, 5.81°), (16.28°, 11.62°), and (24.42°, 17.43°) on the diagonal. To facilitate comparison, the spatial frequency unit was converted to cycles per degree in visual space. The designed peak angular resolution was set to 2 arcminutes per pixel, equivalent to the Nyquist frequency of 15 cycles per degree or 52.1 cycles per millimeter on the retina. Figures 11A–11D present the MTF plots for the Arizona eye model at the aforementioned four angles, where we calculated the average MTF values in the tangential and radial directions for each sampled field angle across the display. The contrast modulation threshold of 10% was used at the maximum angular frequency to determine the perceived limiting resolution. Figures 12A–12D depict the perceived limiting resolution distributions of the perceptual-driven foveated head-mounted display system 700 as a function of the display field angle, considering eye rotation angles of 0°, 10°, 20°, and 30° along the diagonal direction, respectively. Based on the analysis presented in Figs.12A–12D, within the eye movement region of up to 30°, the perceptual-driven foveated head-mounted display system 700 achieves a peak resolution of approximately 50% of a standard observer, corresponding to an angular resolution of around 2 arcminutes per pixel. The perceived limiting resolution of the perceptual-driven foveated head-mounted display system 700 exceeds the visual acuity of a standard observer throughout the entire field of view, with the exception of a small region near the gaze direction. In summary, the perceptual-driven foveated head-mounted display system 700 demonstrates satisfactory perceived performance without the need for dynamic tracking and scanning devices, particularly in areas that undergo frequent eye movements. Figure 13A illustrates the distortion grid of the perceptual-driven foveated head-mounted display system 700 of Fig.7 at the display plane. Figure 13B presents a detailed representation of the resolution distribution across the field angle along the diagonal direction by applying Eq. (7) and ray tracing. To generate a resolution distribution map across the virtual display panel, we traced 800,000 chief rays with a field sampling angle interval set to 0.1% of the full FOV, resulting in a horizontal interval of 0.07 degrees and a vertical interval of 0.05 degrees. To facilitate the ray tracing process in the CODEV^® software, for each certain horizontal sample, we traced the field region vertically from -25° to 25° with a 0.05-degree interval to provide 1000 intervals. From this dataset, we selected the data within our designed FOV ranging from -15° to 25°, yielding 800 chief ray heights at the display plane. It is evident that there is a noticeable variation in resolution distribution as the function of field angle, and the overall view exhibits a high level of rotational symmetry. The simulated degradation rate of resolution is highly consistent with our intended design objective, exhibiting a relatively stable rate within the foveated region and progressively declining at an accelerated pace with increasing field angles. In order to assess the optical performance of the optical see-through path, an ideal lens was introduced 10 mm after the entire system and perpendicular to the optical axis. Figures 14A and 14B present the MTF plot and distortion grid, respectively, of the optical see-through path for the perceptual-driven foveated head-mounted display system 700. The MTF exceeds 40% for all fields at the Nyquist frequency, which corresponds to a resolution of 1 arcminute per pixel for the human eye. Furthermore, the maximum distortion value for the entire see-through view is below 0.1%. Optimization method Freeform surfaces typically provide superior optical performance compared to traditional spherical or aspherical surfaces, which are expressed with a greater variety of parameters and thus offer more degrees of freedom for compensating and balancing aberrations. However, practical designs must take into account design and manufacturing complexities and ultimately the overall cost. Therefore, exemplary overall optical systems in accordance with the present invention maintain symmetry about the YOZ plane instead of being entirely asymmetric, considering the inherent symmetry in the horizontal direction. To demonstrate the process and methods for optimizing a freeform wedge prism with a spatially varying optical power for a perceptual-driven foveated head-mounted display scheme in accordance with the present invention, we used the 3-element sandwich structure 700 shown in Fig.7 as an example. The same process and method is applicable to the 2-element structure 500 shown in Fig.5. As illustrated in Fig.7, the origin of the global coordinate system OXYZ is positioned at the center of the exit pupil, where the eye pupil of a viewer is placed. The Z axis aligns with the head pose direction, the OXY plane is perpendicular to the Z-axis, and the Y-axis is directing upward. It is assumed that the display panel 706 is located above the eyebrow, thus the light ray path for the virtual display is folded within the main freeform prism 702 along the Y direction through two reflections by the surfaces S3’ and S4, respectively. In the design configuration, all optical surfaces are decentered and tilted relative to this global reference. The overall system includes two independent light paths—the virtual display path through first auxiliary freeform prism 705 and main freeform prism 702 and the see-through path via first auxiliary freeform prism 705, main freeform prism 702, and second auxiliary freeform prism 704. However, optimizing the virtual display path is far more complex than the see-through path. Therefore, our design and optimization concentrated on the virtual display path, while the see- through path was separately optimized to obtain the prescriptions of the second auxiliary freeform prism 704, following the completion of the virtual display path design. In the case of no vision correction, the surface S6 of the second auxiliary freeform prism 704 remained flat and surface S4’ of the second auxiliary freeform prism 704 is matched to surface S4 of main freeform prism 702. For this reason, the rest of this section focuses on optimization strategies for the display path. An important aspect of optimizing a freeform system with complex waveguide-like lightpath folding is to define the proper structural and optical constraints. In case of optimizing the dual-element system for the virtual display path, the system design and optimization involve three key aspects: the structural constraints of the prisms 702–705, total internal reflection (TIR) conditions for surface ^^_ଷ ^ᇱ , and optical performance control, particularly for spatially varying optical power as a function of the field angle following a desired foveation scheme characterized by an angular resolution distribution function FFD(θ). For instance, Eq. (4) is an example of a desired resolution distribution function. Structural constraints Since the surfaces S1, S2, S4’, S6 of the two auxiliary freeform prisms 705, 704 are either flat or picked up from the surfaces of the main freeform prism 702, we only need to control the minimum thickness of these auxiliary freeform prisms 705, 704. Considering the overall size of these auxiliary freeform prisms 705, 704, it is important to maintain a minimum thickness of at least 2 millimeters to ensure stability, ease of processing, and practical application. The primary objective of the structural constraints is to ensure the proper propagation of rays within the main freeform prism 702, and the proper formation of a valid freeform prism through its three optical surfaces while maintaining manufacturable center and edge thickness. As shown in Fig.15, the shape of the main freeform prism 702 is governed by the optical paths of the upper marginal ray at the maximum Y-direction field of 25° and the lower marginal ray at the minimum Y-direction field of -15°. The intersection points ^^_^^, ^^_^^, and ^^_^ଶ correspond to the intersections of the upper marginal ray with surfaces ^^_ସ, ^^_ଷ ^ᇱ , and ^^_ହ, respectively. Similarly, the intersection points ^^_^^ିଷ, and ^^_^ଶ correspond to the intersections of the lower marginal ray with surfaces ^^_ଷ, ^^_ସ, ^^_ଷ ^ᇱ , and ^^_ହ, respectively. By tracing the upper and lower marginal rays and locating these points in the global coordinate system, we can establish the following constraints to ensure the physical structure of the prism (Eq. 10), adequate eye clearance (Eq. 11), and maximum thickness (Eq.12): _ì ^{^^^ೌమ − ^^^ೌభ < 0} ^ï _{^^^ೌయ − ^^^ೌమ > 0} where all

system with the origin located at the center of the exit pupil. By applying constraints on the Y coordinates of point ^^_^^ିଷ, the first two equations ensure the proper intersection of surfaces ^^_ଷ, and S4. This allows the lower marginal ray to be traced through the prism without any obstructions. Furthermore, by constraining the Z coordinates of points ^^_^ଶ and ^^_^^, the upper and lower limits (e.g., 5 and 0.2mm, respectively) on the bottom thickness of the prism are established. Controlling the Y coordinates as the fourth equation in Eq. (7), involving points ^^_^ଶ and ^^_^^, ensures the proper intersection of surfaces ^^_ଷ ^ᇱ , and ^^_ହ, allowing the top marginal ray to pass through the prism without obstruction or escape. This control also aids in managing the prism's height. The remaining two constraints in Eq. (10), related to the Y and Z coordinates of points ^^_^ଶ and ^^_^^ prevent the top marginal ray from escaping after reflecting from surface ^^_ସ ^ᇱ and assist in controlling the prism’s thickness. The equations presented in Eq. (10) collectively guaranteed the formation of a valid prism shape with the three optical surfaces. By restricting the Z coordinates of points ^^_^^ and ^^_^^, Eq. (11) provided a minimum value for the eye clearance distance. Eq. (12) further controlled the total thickness of the main freeform prism by constraining the Z coordinates of the points of ^^_^^ and ^^_^^. TIR condition As mentioned earlier, all rays from the entire display FOV are reflected off surface ^^3^′ . Given the substantial overlap between the refractive and reflective optical paths for certain rays via surfaces S3 and ^^3^′, it is not practical to apply a reflective film to ^^3^′. Consequently, it becomes necessary to ensure TIR conditions are satisfied at surface ^^3^′ to maintain correct ray propagation within the prism. The critical angle ^^_^ is determined by the TIR condition, given by ^^_^ = ^^ ^^ ^^ ^^ ^^ ^^( 1/ ^^), (13) where n is the refractive index of the material of the main freeform prism. In this design, Cyclic Olefin Polymer (COP) is utilized instead of Acrylic (PMMA) because of its higher refractive index corresponding to larger a critical angle potentially offering more space for optimization. In the case of design structures starting with spherical or aspherical surfaces, it may be possible to identify a representative ray with the minimum incidence angle across the full field, such as the upper marginal ray for the maximum Y-direction field, to satisfy the TIR condition. However, for freeform surfaces, solely controlling the incident angle of the edge field is not sufficient for TIR to occur accurately for all fields. During the design process, the incident angles of the upper marginal ray, lower marginal ray, and the chief ray of the center field are controlled to be larger than the critical angle ^θ _c . This ensures that all rays across the field of view are reflected. On the other hand, for surfaces ^^3, ^^4 and ^^5, the TIR condition needs to be avoided. In CODEV^®, the MXA function is employed to ensure that the incident angles of all sampled fields are less than the critical angle ^^_^. Field sample In contrast to traditional designs, the statically foveated virtual image in this study exhibits spatially varying magnification characteristics as function of field angle and is not rotationally symmetric. To efficiently control the performance of the system and properly control optical magnification distribution through the FOV, a concentric ring grid field sampling method is employed. Two zoom systems are utilized to sample nine ring grids and several edge fields, as outlined in Table 6, and illustrated in Fig.16, where from left to right the first two arcs (bold stars) represent the sampled fields located within the fovea region, the next five arcs (non-bold stars) represent the sampled fields located within the parafovea region, and the remaining arcs (dashed stars) in the peripheral region. This approach ensures field sampling efficiency and provides enough samples for optimization. Table 6. The sampling field distribution. Zoom1 Zoom 2

F2 0°, -4° 20°, 0° F3 4°, 0° 12°, 17,32° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °

Constraints for spatially varying optical power control Based on the sandwich-structure (main freeform prism 702 between first auxiliary freeform prism 705 and second auxiliary freeform prism 704) of the eyepiece shown in Fig 7, we designed the virtual image optical path by combining the effects of the main freeform prism 702 and the first auxiliary prism 705. The main freeform prism 702 includes three distinctive surfaces, while the first auxiliary prism has a flat surface S1, and surface S2 is picked up from surface S3 of the main freeform prism 702. In the initial design phase, we began with a starting point composed of three conic surfaces or spherical surfaces, which may have a smaller FOV compared to our final target at first. The primary focus of the starting point design was to achieve the correct optical power for the center field while satisfying the structural and TIR condition requirements discussed above. After finding the starting point structure, we converted all the three surfaces into Zernike type surfaces. The resolution distribution of the perceptual-driven foveated head-mounted display system can be determined by analyzing the chief ray heights at the display plane for the sampled fields, as described by Eq. (7). However, due to the freeform nature of the prism surfaces where local optical power can change rapidly, it is inadequate to strictly follow the constraints of the resolution distribution defined by Eq. (7) at critical balance field points, such as the ^^_^^ and ^^_^ଶ which are the critical angles defining the boundaries of the fovea, parafovea, and peripheral regions. Instead, a more robust method needs to be adapted from Eq. (7) to ensure gradual and smooth optical power variation without accidently inducing steep change of optical power by local surface shapes. It is therefore preferable to control the average degradation rate in resolution or average optical power over a region based on the distance (D) at the display panel between the target field θ and the center field. It ensures that the degradation rate changes smoothly and naturally for the three regions of display, avoiding sudden variations. The distance can be expressed as follows: D^{(θ )= ( X ( θ ) − X (0)) 2 + ( Y ( θ ) − Y (0)) 2} _{, (14)}

where the , , , are plane for θ-angle and center field, respectively. This distance can also be converted to represent the cumulative number of pixels utilized, based on the average value of the spatially varying resolution distribution function from the center field to the target field. As discussed above, the fields were sampled as a concentric ring grid with nine rings. Given the non-rotational symmetry of the freeform optics system, it is more efficient to control the average distance value for all fields with the same angular distance from center, which are located at the same ring of the sampled fields. By comparing it with the target resolution distribution value, such as values calculated using Eq. (4), we have effectively controlled the average degradation rate in resolution within the section from the target field to the center field. The constraint for resolution distribution can be expressed as: n ^

where ^^_^( ^^) is the distance at display panel from the center field to the field selected from one of the sampled fields with the angular distance θ as listed in Table 6, and the variable n denotes the total number fields with the angular distance θ located at the same field sampling ring. Another important aspect of optical performance is the control of rotational symmetry in the resolution distribution. This is achieved by controlling the variance of the distances among all the sampled fields with the same angular distance., defined as, n n ^{^} D ⁱ ( θ ) In

by Eqs. (10) through (12), we incorporate the metrics defined by Eqs. (15) and (16) to control the resolution distribution, aiming to gradually align it with a theoretical function such as the one described in Eq. (4) at specific selected fields. We focus on controlling the spatial variation of optical power at angles such as 4°, 15°, and 30°. The selection of these angles is based on two key factors. Firstly, the degradation rate in resolution of the theoretical function around these fields shows relative stability. Secondly, these fields closely represent the average resolution distribution within the fovea, parafovea, and peripheral regions, respectively. By choosing these angles, we ensure a smooth and coherent variation in the resolution distribution, gradually following the desired statically foveated resolution distribution function. Beginning with the initial structure, we initially apply loose constraints on the resolution distribution and symmetry at the 4° field angle. Since the 4° field is within the fovea region and shares a similar resolution distribution with the center field, this step allows us to identify a structure that aligns more closely with our final target while preserving the first-order parameters. Subsequently, the higher-order parameters of the three surfaces were incrementally introduced as variables for optimization. The constraints defined by Eqs. (15) and (16) were gradually incorporated during the optimization process, applied at the field angles of 15° and 30°. Initially, these constraints were implemented with loose controls and were subsequently tightened as the optimization progressed. Additionally, the FOV is also gradually increasing towards the design target. It is important to note two factors in the optimization. Firstly, we minimize the usage of surface ^^5 parameters in the preliminary stage due to its significant impact on aberration control and its role in achieving the desired optical power spatial variation. For example, we optimized the system using 6^th order parameters of the surface ^^3 and ^^4, while only employing a conic type for surface ^^5. This step helps us find a result with good optical performance under relatively loose constraints for resolution distribution and symmetry. Subsequently, we maintain the optics with equivalent performance and gradually introduce the parameters of surface ^^5 as variables, up to 6^th order. This strategy proves to be more time-efficient and facilitates the identification of the appropriate structure compared to direct optimization with 6^th order parameters for all surfaces. Secondly, if the pixel count of the display is sufficient to provide higher perceived performance, we can relax the control requirements for the resolution distribution in each region. Thus, the active size of the display panel serves as an important reference for optimization. Initially unconstrained, it is gradually limited to the actual size listed in Table 3. Ultimately, a total of 60 variables are activated to optimize the system. The constraints for resolution distribution, defined by Eq. (16), are controlled within one pixel, eight pixels, and fifty pixels error for 4°, 15°, and 30° angle field, respectively. These and other advantages of the present invention will be apparent to those skilled in the art from the foregoing specification. Accordingly, it will be recognized by those skilled in the art that changes or modifications may be made to the above described embodiments without departing from the broad inventive concepts of the invention. It should therefore be understood that this invention is not limited to the particular embodiments described herein, but is intended to include all changes and modifications that are within the scope and spirit of the invention as set forth in the claims. References 1. L. C. Loschky and G. W. McConkie, “Investigating spatial vision and dynamic attentional selection using a gaze contingent multiresolutional display,” Q. J. Exp. Psychol. A 8, 99–117 (2002). 2. W. Zhou and A. C. Bovik, “Embedded foveation image coding,” IEEE Trans. Image Process. 10, 1397–1410, 2001. A. T. Duchowski and A. Çöltekin, “Foveated gaze-contingent displays for peripheral LOD management, 3D visualization, and stereo imaging,” ACM Trans. Multimedia Comput. Commun. Appl.3, 1–21 (2007) G. Sandini, et al, “A retinalike CMOS sensor and its applications,” in Proceedings of IEEE Workshop on Sensor Array and Multichannel Signal Processing (IEEE, 2000), pp.514–519. D. V. Wick et al, “Foveated imaging demonstration,” Opt. Express 10, 60–65, 2002. H. Hua and S.Liu, “Dual-sensor foveated imaging system,” Applied Optics, 47(3), 2007. J. P. Rolland, et al, “High resolution inset head-mounted display,” Appl. Opt.37, 4183–4193, 1998. Tan G, et al Foveated imaging for near-eye displays. Optics express, 26(19), 2018. K. Iwamoto, S. Katsumata, K. Tanie, “An eye movement tracking type head mounted display for virtual reality system: -evaluation experiments of a prototype system”, Proceedings of 1994 IEEE International Conference on Systems, Man, and Cybernetics. Humans, Information and Technology (Cat. No.94CH3571-5). vol.1, pp.13-18, 1994. P. Lyu and H. Hua, "Perceptual-driven approach to statically foveated head-mounted displays." Optics Express, 29(21), 33890-33914, (2021). Burgess-Limerick R, Mon-Williams M, Coppard VL. Visual display height. Human Factors. 2000 Mar;42(1):140-50. Cook G, Stark L. Derivation of a model for the human eye-positioning mechanism. The bulletin of mathematical biophysics.1967 Mar 1;29(1):153-74. W. S. Geisler and J. S. Perry, “Real-time foveated multiresolution system for low-bandwidth video communication,” Proc. SPIE 3299, 294–305 (1998). L. C. Loschky, G. W. McConkie, H. Yang, and M. E. Miller, “The limits of visual resolution in natural scene viewing,” Visual Cognition 12, 1057–1092 (2005). D. Cheng, Y. Wang, H. Hua, and M. M. Talha, "Design of an optical see-through head- mounted display with a low f-number and large field of view using a freeform prism," Appl. Opt.48, 2655–2668 (2009). X. Hu and H. Hua, "Design and tolerance of a free-form optical system for an optical see- through multi-focal-plane display," Appl. Opt.54, 9990–9999 (2015). 17. D. Cheng, J. Duan, H. Chen, H. Wang, D. Li, Q. Wang, Q. Hou, T. Yang, W. Hou, and D. Wang, "Freeform OST-HMD system with large exit pupil diameter and vision correction capability," Photonics Res.10, 21–32 (2022). Various citations, patent applications/publications, and references are cited herein, each of which is incorporated herein by reference in their entirety.

Claims

CLAIMS What is claimed is: 1. A perceptual-driven foveated head-mounted display system for use in creating an image of a display panel that has a uniform pixel density, the system comprising: a wedge-shaped freeform prism having a spatially varying optical power as a function of a field angle wherein a virtual image of the display panel created by the wedge-shaped freeform prism has an angular pixel density that decreases with an increase of field eccentricity.

2. The perceptual-driven foveated head-mounted display system of claim 1, wherein the spatially varying optical power, ΦEP, of the wedge-shaped freeform prism is given by ௧^^( ^భ = _లబ⋅ಷ ) ಷ_ವ(ഇ) , where p₀ is the pixel pitch

pixel density, θ is the field angle of a pixel on a plane of a virtual image as seen by a viewer through the wedge-shaped freeform prism, and FD is the angular pixel density in the virtual image.

3. The perceptual-driven foveated head-mounted display system according to any one of the preceding claims, wherein the angular pixel density, F_FD(θ), in the virtual image is given by ^_^ி^ ⁽ _^^ ⁾ ₌ ^{^} )_, where p0 is the

density, θ is the field angle of a pixel on a plane of a virtual image as seen by a viewer through the wedge-shaped freeform prism, and _{Φ EP} is the optical power of the wedge-shaped freeform prism.

4. The perceptual-driven foveated head-mounted display system according to any one of the preceding claims, comprising a see-through auxiliary freeform prism disposed in optical communication with the wedge-shaped freeform prism at a surface thereof different from a surface of the wedge-shaped freeform prism proximate the display panel.

5. The perceptual-driven foveated head-mounted display system according to claim 4, wherein the wedge-shaped freeform prism and see-through auxiliary freeform prism cooperate to provide a real-world scene along a see-through path disposed through the wedge-shaped freeform prism and see-through auxiliary freeform prism.

6. The perceptual-driven foveated head-mounted display system according to any one of the preceding claims, comprising a front auxiliary freeform prism disposed in optical communication with the wedge-shaped freeform prism to provide a virtual display path therethrough extending from an exit pupil of the system to a first surface of the wedge- shaped freeform prism configured to be disposed proximate the display panel.

7. The perceptual-driven foveated head-mounted display system according to claim 6 as it depends from any one of claims 4–5, wherein the wedge-shaped freeform prism is disposed between the front auxiliary freeform prism and the see-through auxiliary freeform prism and wherein the front auxiliary freeform prism is disposed closest to the exit pupil.

8. The perceptual-driven foveated head-mounted display system according to any one of claims 6–7, wherein the front auxiliary freeform prism includes a first surface proximate the exit pupil, the first surface having an optical power to correct myopia or hyperopia of a user of the system, whereby the front auxiliary freeform prism provides a vision correction lens.

9. The perceptual-driven foveated head-mounted display system according to any one of claims 6–7, wherein the front auxiliary freeform prism and the see-through auxiliary freeform prism each have a respective flat surface, the flat surfaces disposed parallel to one another.

10. The perceptual-driven foveated head-mounted display system according to any one of the preceding claims, wherein the wedge-shaped freeform prism includes a first, second, and third optical surface at which light is refracted and/or reflected, the first optical surface configured to be disposed to receive light from the display panel, the second and third optical surfaces being relatively longer than the first optical surface, wherein at least one of the first, second, and third optical surfaces is a freeform optical surface and the first, second, and third optical surfaces are disposed relative to one another to provide the wedge-shape of the wedge-shaped freeform prism.

11. The perceptual-driven foveated head-mounted display system according to claim 10, wherein the first, second, and third optical surfaces are all freeform optical surfaces.

12. The perceptual-driven foveated head-mounted display system according to claim 10 or 11, wherein one or more of the first, second, and third optical surface(s) is a Zernike surface of the form

, each Zernike surface having a finite, non-zero radial distance, r, and at least one of the first, second, and third optical surface(s) having one or more finite, non-zero Zernike polynomial coefficients, C_J+1, that provide the spatially varying optical power, Φ_EP, as a function of a field angle.

13. The perceptual-driven foveated head-mounted display system according to any one of claims 10–12, comprising a global coordinate system OXYZ wherein an origin of the global coordinate system OXYZ is positioned at the center of an exit pupil, a Z axis aligns with a head pose direction of a user, a OXY plane is perpendicular to the Z-axis, and a Y-axis is directing upward, and one or more of the first, second, and third optical surface(s) is symmetric about a YOZ plane perpendicular to an X-axis of the global coordinate system.

14. The perceptual-driven foveated head-mounted display system according to any one of the preceding claims, comprising a display panel with uniform pixel density disposed in optical communication with the wedge-shaped freeform prism.