JP2020512735A - Compression method and system for near-eye display - Google Patents

Abstract

An image compression method for a near-eye display system that reduces input bandwidth and system processing resources is disclosed. Combined with the use of high-order basis modulation, dynamic color gamut, optical field depth sampling, and compressed input displays that provide image data word length reduction and quantization for the purpose of matching the angles of the human visual system. Color and depth vision enable a high fidelity visual experience in near-eye display systems suitable for mobile applications with substantially reduced input interface bandwidth and processing resources.

Description

  The present invention relates generally to compression methods for imaging systems, and more particularly to compression methods for imaging systems and image and data compression methods for head-mounted or near-eye display systems. These are collectively referred to as a near-eye display system in the present specification.

  Recently, near-eye display devices have received widespread attention. Near-eye display devices are not new, and many prototypes and commercial products can be traced to the 1960s. DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention However, recent advances have been made in networked computing. Advances in embedded computing, display technology and optical design have renewed interest in such devices. Near-eye display systems are usually combined with a processor (embedded or external), a tracking sensor for data collection, a display and the necessary optics, and the processor typically processes the data acquired from the sensor. And generate data to be displayed as a virtual image within the field of view of one or both eyes of the user. This data can range from simple warning messages or 2d info charts to complex floating animated 3d objects.

  Recently, the class of display for near-eye has attracted attention. That is, near-eye augmented reality (AR) virtual reality (VR) is displayed as a next-generation display that presents a viewer with a "life-like" visual experience. In addition, near-eye ar displays extend what is seen as a last resort for presenting mobile viewers with high-resolution 3d content t to the viewer's ambient reality scene, extending access to the viewer's information. The primary purpose of Ar displays configured to convey the viewing restrictions of current mobile displays is to provide viewing ranges that are not limited to the physical restrictions of mobile devices without compromising the mobility of the user. That is. On the other hand, near-eye vr displays are expected to present a viewer with a 360 ° 3d movie viewing experience that immerses the viewer in the viewed content, both Ar and vr display technologies for mobile phones and personal computers. Seen as the "next computing platform" behind the continuum of, AR / vr displays are often used herein to expand the growth of mobile users' information access and the growth of the information market and the businesses that serve it. Called a "near-eye" display, it emphasizes the fact that the method of the invention is generally applied to near-eye displays and is not limited to ar / vr displays per se.

  The main drawbacks of existing near-eye ar and vr displays include motion sickness caused by low refresh rate display technology; avoiding eye distortions and nausea caused by, and limiting eye in a reasonably wide field of view (FOV). Allows to achieve the resolution achieved. Existing attempts to solve these shortcomings have involved using a display with a higher refresh rate, using a display with more pixels (higher resolution), or using multiple displays or image planes. A common theme during all these attempts, including using, is the need for higher input data bandwidth. Allows higher data bandwidth to be accommodated without adding size, complexity and excessive power consumption. [Solution]. While the use of compression is the usual solution for dealing with large amounts of data, the requirements of near-eye displays are unique and can be achieved by traditional video compression algorithms. Video compression for eye displays must achieve higher compression ratios than provided by existing compression schemes, with the added requirement of very low power consumption and low latency.

  The near-eye display's high compression ratio, low latency, and low power consumption constraints are similar to data compression schemes that utilize such human visual system (HVS) features that require a new approach to data compression (t). SUMMARY OF THE INVENTION DISCLOSURE OF THE INVENTION PROVIDING COMPRESSED CAPTURE AND DISPLAY AND COMPRESSED CAPTURE AND DISPLAY The present invention therefore seeks to overcome Introducing a method, in this way, makes it possible to create a near-eye display that can meet stringent mobile device designs. The compactness and power consumption requirements and users of such devices enhance the visual experience of either 2d or 3d content over a wide angular range. Further objects and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments with reference to the accompanying drawings.

  There are numerous prior art describing methods for near-eye displays. Typical examples are maimon, Andrew, and manyfuchs. "Computational Augmented Reality Glasses." Mixed augmented reality (ISMAR)) eyeinternational symposium, pp Computational Augmented Reality (AR) display. The present invention does not address the data compression and low latency requirements of the described near-eye display prototypes that utilize LCDs to reproduce brightfield through stacked layers. The ar display also has a wide field of view, achieving a non-encoding format that allows for occlusion of each other and depth of focus cues. However, the process for determining the LCD layer pattern is based on computationally intensive tensor factorization which is very time consuming and power consuming. This ar display also has significantly reduced brightness due to the use of light blocking LCDs. This is how the display technology affects the performance of the near-eye display and how the prior art will be shortened in solving the problems presented by the near-eye display area. This is another example.

  An eye comprising a combination of elements such as a processor, which may be the exemplary prior art near-eye display system (100) shown in FIGS. 1a and 1b, an embedded processor (102) or an external processor (107). And head tracking element (210), display device (103), and optics (104) for expansion and relay of the displayed image to the human visual system (HVS) (FIG. 1a) or 107. This data processing, characterized in that it processes perceptual data obtained from the eye and head tracking elements (210), which are any of (FIG. 1b), to produce the corresponding image to be displayed. Internally in a near-eye device with 102 (FIG. 1). Such processing can be done remotely by an external processor (107) (FIG. 1b). The latter approach is more like a task-specific processing unit for processing the latest generation CPUs, GPUs, and incoming traces. Enable the use of more powerful processors. A more powerful image with the processing throughput and memory needed to handle the image processing without burdening (109) the conversion of the data into data and the transmission of the corresponding image over the personal area network (PAN). There is the advantage that a remote processing device (107) can be used. On the other hand, sending data over the pan has its own challenges, such as low latency, high resolution video transmission bandwidth requirements. New low-delay protocol for video transmission protocols in pan (see zravi, R; Fleury, M; Ghanbari, M.), "Low-delay video control in personal area networks for augmented reality", image processing , IET, vol. 2, no. 3, pp. 150-162, June 2008), which allows the use of an external processor (107) for near-eye display image generation to enable high quality immersive stereoscopic vr display. Such pan protocols cannot address the high bandwidth data requirements for new generation of near-eye ar and vr displays aimed at presenting viewers with high resolution 3d and VAC-free viewing experience. .

  Nevertheless, the use of more advanced display technologies poses new challenges to the overall system. New imaging methods increase the amount of data generated and need to be sent to the display, due to size constraints, traditional compression methods used to handle expanded amounts of data are No longer suitable. Therefore, a new way to generate, compress and send data to near-eye displays is needed

  In the description below, like reference numbers are used for like elements in different drawings. A comprehensive understanding of the exemplary embodiments is provided given the items defined in the description, such as detailed configurations and elements. However, the present invention can be implemented without using any particular limitation. Also, well-known functions or constructions are not described in detail as they do not obscure the present invention in unnecessary detail. In order to understand the present invention and how it can be implemented, some embodiments of the present invention are described by way of non-limiting examples with reference to the accompanying drawings.

FIG. 1 is a block diagram of a prior art near-eye display system incorporating an embedded processor. FIG. 3 is a block diagram of a prior art near-eye display system incorporating an external processor connected. FIG. 6 is a block diagram of the near-eye display system of the present invention with an embedded processor. FIG. 6 shows a block diagram of a near-eye display system of the present invention with an external processor. FIG. 6 shows a functional block diagram of an encoder that applies a visual decompression feature of a compressed display within the context of the near-eye display system of the present invention. 6 illustrates a base coefficient modulation of the visual decompression method of the present invention. 6 illustrates the base coefficient truncation of the visual decompression method of the present invention. FIG. 6 is a diagram showing a field of view (FOV) region around a viewer's point of gaze used by the foveated visual decompression method of the present invention. FIG. 3 shows a block diagram of a near-eye display system incorporating the foveated visual decompression method of the present invention. FIG. 6 is a diagram showing the base coefficient truncation of the foveated visual decompression method of the present invention. 5 shows an implementation of a light modulator element of a near-eye display system that matches the viewer's HVS angular velocity and FOV. 3 shows an implementation of the optical elements of the near-eye display system of the present invention. 1 illustrates a multi-focal plane embodiment of this near-eye optical field display of the present invention. 1 illustrates an embodiment of the present invention. This embodiment uses a standard horopter surface to achieve a multifocal plane near-eye display. FIG. 6 illustrates the generation of content for a multi-focal plane near-eye light field display of the present invention. FIG. 6 illustrates an embodiment for implementing the multi-focal depth filtering method of the present invention. FIG. 5 illustrates an embodiment implementing compressed rendering of light field data input to a multi-focal plane near-eye light field display of the present invention.

Reference in the following detailed description of the invention
"One embodiment" or "one embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearances of the phrase "in one embodiment" in various places in this detailed description are not necessarily referring to the same embodiment.

  Presenting a near-eye display viewer with a high resolution, wide field of view (FOV) 3d viewing experience requires a display resolution approaching the viewing limit of 8 megapixels per eye. The resulting increase in display resolution overall imposes some requirements on the near-eye display system. The most difficult of them is the increase in data interface bandwidth and processing throughput. The present invention introduces a method for addressing both of these challenges in near-eye display systems by using a compressed display system (as defined below). 2a and 2b are block diagrams, a near-eye display system (200) using the method of the present invention. In FIG. 2a, responsible for compressing data for a compressed display (203), illustrating one embodiment of the near-eye assembly (201), new design element, encoder (204) of the near-eye display system (200). A QPI solid-state imager-based display (QPI imager display in the figure), such as a QPI solid-state imager-based display (QPI imager display in the drawing), added to the near-eye display system (200), eg (7 , 767,479 and 7,829,902). Solid-state LED or laser with each pixel having a different color. A laser that serves multiple solid state LEDs or a single pixel in addition to a QPI imager that emits light from a stack of emitters. Solid-state LEDs or lasers of different colors arranged side by side with emitters. Imaging devices that emit light from an emitter are also known. Such devices of the present invention are commonly referred to as light emitting display devices. In addition, the present invention also finds use as a backlight source for LCDs, such as many types of spatial light modulators (SLMs, which can be used to create light sources for microdisplays) dl and LCOS. be able to. As used herein, solid-state imager display, display element, display, and like terms are used herein to refer frequently to a compressed display (203), FIG. 2D shows another embodiment of the near-eye assembly (205) of the near-eye display system (200), the encoder (204) performs the same function as in FIG. 2a, but with an external data source that drives the near-eye assembly (205) remotely. Acting as part FIG. 2b is a wireless showing an external data source that includes an external processor (207), the latter of which has an encoder (204) connected to t. personal. area. Wireless such as network (PAN). Near eye display via link (208). The encoder (204), in both cases (transmitting to the assembly (205)) or connected via wires (209), utilizes the compressed processing power of the solid state imager display (203), A high compression rate is realized while generating a high quality image. The encoder also utilizes the perceptual data provided by the eye and head tracking design elements to further increase the data compression gain of the near-eye display system.

Definition −
"Compressed (input) display" -is a display system, a subsystem capable of directly displaying a content image of provided compressed data that is directly input in a compressed format without first decompressing the input data. Provide the element. Such compressed displays can modulate images at high subframe rates based on high order for direct perception by the human visual system (HVS). Such a display function, referred to as "visual stretching", defined below, allows a compressed display to modulate a high order macro containing (nxn) pixels using the expansion coefficient of the Discrete Cosine Transform (DCT). Or Discrete Walsh Transform (DWT) is applied directly to the HVS to be integrated and perceived as a decompressed image. (U.S. Pat. No. 4,970,646)

"Dynamic gamut" -Compressed display systems may also include a feature known as dynamic gamut (US Pat. No. 4,524,682)) word length adjusted (compressed) gamut data provided in the frame header. A display system in which the color gamut can be dynamically adjusted for each frame is provided using. When using the dynamic gamut capability, the compressed display system processes the input data, using the compressed gamut and HVS visual acuity that match the input data to the gamut of the input frame image, Convert to the corresponding image. Visual stretching and dynamic so input data need not be needed. Both gamut features reduce interface bandwidth and display-side processing throughput
For example, a compressed display such as a solid state imager display decompresses and supports both features.

  "Visual Decompression" -a number of compressed uses the inherent perceptual power of HVS to allow modulation of the visual information compressed directly by the display, rather than first decompressing and displaying the decompressed visual information. It is a visual information modulation method. Visual decompression reduces the interface bandwidth to the display and the processing throughput required to decompress compressed visual information.

  Visual Decompression-FIG. 3a shows the functional block diagram of the encoder 204 (FIGS. 3a and 3b), applying the visual decompression function of the compressed display (203) within the context of the near-eye display system (200) of the present invention. To do. The input image (301) generated by the processor (202) or (207) is transformed into a known higher order basis, eg DCT or DWT, eg by a visual stretch transform element (302). The selected subset of resulting coefficients of these higher order bases is then quantized by a quantizer. It is then applied by the encoder (204) of the inventive near-eye display system (200), which is quantized by a quantizer, similar to typical compression schemes using DCT and DWT such as MPEG and JPEG. Visual stretching achieves partial compression gain by selecting a subset of the bases with low frequencies while truncating the high frequency base. In one embodiment of the invention, the quantizer uses the same quantization step size to quantize the selected subset of basis coefficients. In another embodiment of the invention, the quantizer uses high frequency coefficients to reduce the data transfer bandwidth associated with coefficients that are not perceivable by HVS (which utilizes the capabilities of the human visual system (HVS)). Use a larger quantization step for. Therefore, by matching the HVS function, a higher visual stretch gain can be achieved. The quantized coefficients are visually decompressible compressed into a set of coefficients associated with one of the selected bases that is time (or time division) multiplexed by a run length encoder (304). The compressed display (203), which transmits to the display (203) at a time, modulates the coefficients it receives as the size of the associated basis macros it displays. The compressed display (203) has a modulation base of HVS impulse response. Modulate one based on time within one video subframe so that it is not temporally separated by more than a time constant, eg, 60 Hz if 8 bases are selected to transform the input image (302). A (3.67 Ms) video frame is divided into -2 ms subframes, which are timed in the HVS impulse response. Well below one base coefficient is modulated by the display (203) that is compressed.

  In another embodiment, the visual decompression transform block supplies the extracted DWT and DCT coefficients to the quantizer (303), which extracts the DWT and DCT coefficients directly from the externally provided compressed input data format. , Quantizer (303) further uses a larger quantization step for high frequency coefficients to increase the DWT and DCT coefficients of MPEG and JPEG data formats and to match the HVS function for higher visual performance. In order to achieve a typical stretch gain, the coefficients that are not recognized by HVS are regenerated.

  In another embodiment of the invention, the transformed (302) and quantized 303 input image (301) basis coefficients are field-ordered directly into (50t) (304) visually compressed data. To provide a compressed display (203) that can be directly modulated into HVS (see Predefinition of Compressed Display) to reduce the memory requirements on the display (203) in order to achieve visual decompression gain. In addition, the method of direct transfer and modulation of compressed image data reduces the latency of transferring image data from the processor (202) or (207) to the display (203) and to the HVS 106. In the eye, the method of direct transfer and modulation of compressed image data, the subset of basis coefficients is typically HV. When received in a temporal sequence of subframes shorter than a time constant, the latency is reduced because it is continuously modulated by the display (203) to the HVS 106, that the HVS 106 begins to partially integrate them. , And allows for gradually recognizing the image input (301) in some of the subframes of the modulated basis coefficient, thereby substantially eliminating the feedback delay in incorporating the gaze direction information sensed. The head tracking (210) in the input image (301). The compressed input image (301) also has a latency as well as a direct transfer and modulation method of the compressed image data as represented by the selected basis coefficients generated by the encoder (204). The HVS 106 is reduced, but without the processing delay typically introduced by conventional systems in which the data of the input image (301) is first compressed on the processor (102) or 107 side and decompressed on the display 203 side. It is displayed directly. In addition to reducing the latency of a near-eye display system, the described near-eye visual decompression method of direct transfer and modulation of compressed image data of the present invention also substantially reduces processing, input image (301). The processing related to the compression of (1) is canceled by either the processor (102) or the 107 side, and the display 203 side is decompressed. The present invention is valuable in providing the near vision visual decompression method described for the direct transfer and modulation of t compressed image data. By taking advantage of the inherent capabilities of HVS 106 for perceptual integration, the present invention reduces latency and processing requirements. That is, the described near-eye visual decompression method of direct transfer and modulation of compressed image data of the present invention achieves reduced latency and processing requirements by matching the capabilities of HVS.

  FIG. 3b shows a visual basis coefficient modulation. The defrosting method of the near-eye display system of the present invention. Each display pixel instead of the row / column selection method instead of the row / column selection method typically used in current displays for addressing (and modulation)) Modulates a group of (nxn) pixels representing a high dimension with the same basis coefficient value ctj in the near-eye visual expansion method shown in FIG. 3b. Within a sub-frame of a video input image (301), a near-eye compressed display (203) with pixels as macros representing display basis elements with corresponding basis coefficients q (addressing blocks of (nxn)). The temporal sequence of basis coefficient modulated sub-frames within a video frame are temporally consecutively integrated by the HVS which is guided to gradually perceive the input image (301) within the time period of the video frame. As can be seen in FIG. 3B, the near-eye compression display (203) has eight subframes, which must possess response time and modulation capabilities to receive and modulate the base coefficients at the subframe rate. In one example, the visual decompression sub-frame rate would be 8 × 60 Hz = 480 hz. In one embodiment of the invention, a near-eye compression display is implemented using a solid state imager due to its high speed image modulation capability. It In addition to the solid state imaging device's ability to support vision, the decompression method of the present invention allows the near-eye display system of the present invention to benefit from small size as well). , QPI 203 provides low power consumption and brightness. Volumetrically simplified near-eye display system.

Referring back to FIG. 3a, the quantizer is based on t, based on a given truncation criterion that truncates the basis coefficients calculated by the visual stretch transform element, and based on a given quantization criterion. , Quantize a selected subset of basis coefficients to a given word length. FIG. 3c shows the basis coefficient truncation performed by the quantizer (303) as a (4 × 4) visual decompression criterion. As shown in FIG. 3c, the quantizer truncates the set of 16 basis coefficients by selecting the subset of 8 basis coefficients marked in FIG. 3c. The criterion for this choice is to discard the high frequency basis coefficients that exceed the HVS temporal limit of vision, and in FIG. 3c a higher index-based hatched basis coefficient selection was chosen. For the subset, the quantizer then truncates the corresponding word length received from the visual decompression transform into a smaller number of bits, eg, an 8-bit word. Note that visual thawing is visual thawing. The transform (302) typically performs the transform computation on a higher word length, eg, a 16-bit word. In another embodiment, the quantizer (303) truncates the selected subset of basis coefficients using different word lengths for different basis coefficients. For example, a description will be given with reference to FIG. 3C. The low frequency coefficient C ₀₀ is quantized to 8 bits, and the remaining basis coefficients along the row coefficient C _oj and the column coefficient C _i0 are quantized using successively lower word lengths, eg 6 bits. 4 bits and 2 bits. Both the truncation of basis coefficients and the quantization criterion of their word lengths are fixed a priori by the display (203) and are either known or signaled (communicated) via the display. It is transmitted to the display (203) embedded in (203). The data transfer bandwidth compression gain expected to be achieved by the near-eye visual decompression method of this embodiment is typically the basis coefficient used to transform the input image (301) depending on the dimension of t. Including a truncation criterion and a basis coefficient truncation criterion used by the quantizer (303), typically in the range (44-6x), from the processor (102) or (107) to a display element ( It means that the image data transfer bandwidth to 203) is reduced by a factor in the range (4x) to (6x) by the described visual decompression method of this embodiment. It should be noted that the visual compression gain embodiment of this is achieved by allowing the display to match the HVS temporal vision.

  "Dynamic Gamut" -In another embodiment of the present invention, the near-eye display system utilizes the following factors to provide additional visual decompression opportunities: (1) The color gamut of a video frame is typically The color coordinates of display pixels within a standard color gamut, which is much smaller than the set standard display color gamut, for example NTSC, is typically represented by a 24-bit word of 8 bits per color primary. And ((2) the visual acuity of the HVS peripheral area is substantially reduced compared to the visual central area. In this embodiment, the visual decompression transform block includes a frame color transform within each input video frame header. The color coordinates of the gamut primary colors are calculated, and the color coordinates of each pixel in the frame expressed with respect to the frame gamut primary colors carried in the frame header are calculated, and the received frame header is passed forward to the quantizer block. (302) passes the frame gamut header it receives to the quantizer block (303) along with the set of higher-order basis coefficients it has extracted. The quantizer block (303) represents each pixel in the image frame. Reduces the size of the image frame gamut by proportionally truncating the word length, which represents the color coordinates of, to less than the default 24 bits (8 bits / 8 bits per color) Can be used to represent the color coordinates of each pixel in each received image frame using a value that is smaller than the transmitted frame color gamut size relative to the display standard color gamut size and smaller than the default 24-bit word length. The visual decompression block (302) is also capable of receiving gamuts and gamuts within each input video frame header, where the coordinates of multiple image areas within an image frame are defined for that frame image area. The color coordinates of each pixel in each frame image area represented with respect to the gamut primaries carried in the frame header are determined, as well as the coordinates of the plurality of image areas in the image frame, in this case the quantizer block (303). ) Is within the range of proportionally truncating the word length representing the color coordinate of each pixel (50t). For a typical video frame image that is less than 8 (8 bits / color), any of the (2) methods described will add a factor of (2x-3x) to a size of 2x-3x of (2t). Image frame data that can lead to a reduction has to be transferred to the compressed display (203), the latter method achieving a compression ratio closer to the higher end of the range. The gamut is received by a compressed display (203), as defined above, the compressed display (203) being carried within one t if it has the ability to adjust its gamut dynamically. The received header to synthesize the gamut of the received frame or frame subregion using the gamut coordinate data of the frame or frame region using its original primaries. Data is received and represents the received (truncated) frame or frame sub-region pixel, and the color coordinate data for modulating the light represents each frame or frame sub-region pixel. The visual compression gain of this embodiment is realized by matching the display color gamut with the image frame color gamut.

  "Foveated Visual Decompression" -This embodiment illustrates yet another method of visual stretching of the near-eye display system (200), as shown in Figures 4a and 4b (a). Calculate and track observer's gaze direction (axis) (401) and focal length based on observer interpupillary distance (ipd), then displayed in observer's field of view (FOV) Used to apply different visual stretch basis coefficient truncation and quantization criteria to different regions of the image) (420), which allows the observer's eye to focus (402) while utilizing the HVS angle (visual acuity). The distribution of visual perception to achieve a high level, which effectively enables the highest visual perception within an FOV region, visual compression is the FOV 403-of a viewer whose HVS visual acuity gradually decreases. 412's It is systematically performed over a region of Ri. In this embodiment, visual stretching is applied to match the angular distribution of HVS visual acuity using a compressed word length that is proportional to the angular distribution of the viewer's vision across the FOV.

FIG. 4a illustrates the method of visual stretching of this embodiment, which is referred to as "foveated visual decompression" and utilizes the viewer's spatial (angular) facts. To achieve a higher visual decompression gain than in the region where the observer's eye is in focus at the focal point 402 (fossa region of the retina) (FOV 403-412). Pafaovea 403-406 and systematic reduction over the rest of the perforea region of the retina (407-412) In this embodiment, the viewer's eye focus and gaze direction 401 cues are fovealized quantization. 430 extracted by the instrument and the perceptual data provided by the eye and head tracking element (210) sensor, for example, the gaze direction of each eye is Determined by the position of each eye pupil within the reference head direction frame detected by the head tracking element (210) sensor, similarly defined as the distance at which both observer eyes are focused and converged. , The near-eye display system observer focal length (or movement distance is also defined) determined by the relative interpupillary distance (IPD) between the centers and by the eye and head tracking element (210) sensor. Determined by the relative interpupillary distance (IPD) between the centers of the detected (two) pupils of the observer. Due to the region of the FOV 420 that the observer focuses on, the maximum likelihood quantizer (430), which typically covers the fossa region of the retina when focused by the observer's eye lens, has a basis coefficient of The highest image resolution is achieved by choosing a large possible subset and using the largest possible word length to quantize the selected subset of basis coefficients. For the remaining regions of the viewer's FOV 420, in the region of FIG. 4a (403-412), the foveated quantizer selects a smaller subset of basis coefficients and quantizes the selected basis coefficients. Use a smaller number of bits to implement In applying such basis coefficient truncation and quantization criteria, embodiments that apply such basis coefficient truncation and quantization criteria achieve the highest resolution within the observer's focal region, It achieves a systematically low resolution over the remaining regions 403-412 of the observer's FOV 420, and a higher visual stretch gain over those FOV regions, without compromising perception. It should be noted that the term "foveated" used within the context of this embodiment is meant to indicate that the display resolution is fitted to the HVS visual acuity profile (distribution). Note that it is configured to move outward from the center of the eye of the observer toward the peripheral region of the retina of the observer's eye. Such image resolution depending on the viewer's line-of-sight direction is described in B, Finch, M, drckker, s, Tan, d, known in the art as "foreground rendering". , And nyder, J, "FOVelated3dgraphics", acmsiggraphasia, 11, 2012, processor (102) or (107), typically centered on image input (301) by image rendering. A band of an image interface (301) in a display (203) that can be achieved by the described FOVelatedvisualdecompression method of the present embodiment, which potentially reduces the image rendering computational load in. Not be converted directly to the reduction of the width and decompression calculation load

  FIG. 4b shows a block diagram of a near-eye display system using the foveated visual decompression method of the present invention. Referring to FIG. (4), by knowing the focus of the viewer based on the input provided by the eye and head tracking elements (210), a fovealization is performed after the visual decompression transformation (302). Selected quantizer (430) is present and the displayed image area corresponds to the observer's focus area 402 (image area focused on the eye). The highest spatial resolution (which is adapted to be projected to the foveal region of the retina), while the rest of the viewer's FOV 420 (403-412) has a systematically lower resolution. Permeability of the retina of the observer, which is configured such that the gradation of visual acuity (in space) of the eye of the person having (or being proportional to) varies across the paraphono vea. FIG. 4c is a decompression method of the present invention showing an example of base truncation and quantization selection of a foverated quantizer according to foveated vision. FIG. 4c shows an example of the basis coefficient truncation performed by a quantizer (430) that has been foveated for a (4 × 4) foveated visual stretch criterion. As shown in the example of FIG. 4c, the foveated quantizer corresponds to the focal region (402) of the viewer truncating the set of 16 basis coefficients by selecting the coefficient t. For that region (402) that produces the largest subset of (8) basis coefficients marked in the first panel of figure (4c), the foveated quantizer (430) also has the highest quantum. The digitized word length, eg, 8 bits per color, is used to represent the selected basis coefficients for the region (402) of the viewer's FOV. As shown in FIG. 4c, for the peripheral focus region, the foveated quantizer transforms the set of 16 basis coefficients into a smaller subset of the 7 basis coefficients marked as shown in FIG. 4c. For that region to be truncated, the foveated quantizer also selects a shorter word length, eg, 7 bits or 6 bits, to represent the selected basis coefficients for the region of the viewer's FOV. be able to. As shown in FIG. 4c, because of the peripheral region, the foveated quantizer produces a set of 16 basis coefficients, systematically better than the basis coefficients, as marked in FIG. 4c. It is also possible to select a short word length, eg less than 6 bits, to truncate to a smaller subset to represent the selected basis coefficients for the region of the viewer's FOV.

  Referring back to FIG. 4b, the control data yielding the truncated quantized basis coefficients produced by the foblated quantizer for multiple FOV200 regions t. Packet (or data. Run length to embed header. Further encoded by encoder (435)) Code including signal (or identifying) contained in streamed data and its truncation and quantization word length A run-length encoder (435), eg, in the data field designated for transmitting the basis coefficient value q, included in the encoded data stream determines whether the basis coefficient value cij is included. A header containing the data field to be played and its associated quantization word length. The added basis coefficients are then time-compressed to the compressed display (203) for one of the coefficients for one of the selected bases as a set of time division multiplexed coefficients. The compressed display (203) is then sent to the compressed header (203) and the control header added by the run-length encoder (435), which decodes its coefficient t, is accordingly sized accordingly. Modulates the received coefficient. As shown in FIG. 4C, the number of basis coefficients associated with the display area (403-412) is systemically reduced, the displayed image resolution is proportional to a typical HVS visual acuity distribution, It is systematically reduced over these areas of the displayed image. As explained above, the criterion for selecting the basis coefficients contained in each of the display regions 403-412 determines the visual acuity of their corresponding retinal regions (based on angle (spatial)), which criterion is: Set as a design parameter of the frustrated quantizer (430)

  The data transfer bandwidth compression gain that is expected to be achieved by the near-eye foveated visual stretching method of the present invention is generally the basis dimensionality used to transform the input image and the foverate. Depending on the base coefficient truncation and the quantization criterion used by the quantized quantizer, it typically exceeds that of the visual decompression method described above. Once the eyes are in focus, the displayed image area nominally spans the angular range of the FOVea area (approximately 2 °)). A display unit that displays the image for the right eye displayed on the display unit and the display system (200) displayed on the display unit have a total FOV of 20 °, for example, a book having a fovealized visual. The decompression method of the invention achieves a compression gain in the range of 4x-6x in the displayed image areas, and when using the example of basis coefficient truncation shown in FIG. 4C, between the displayed image areas. In achieving a systematically high compression gain, the achieved compression gain increases by 8/7 for regions (403), (404), and 8 / for regions (405), (406), respectively. Constructed with a factor of 5, 8/3 Considering the area of each image area (401-412) with respect to the displayed image FOV being formed, and the overhead due to the control data added by the run-length encoder (435), the peripheral area (407) To (412) to (412) are used, the composite compression gain that can be achieved by the Foveed visual decompression method of the present invention is in the range of (24x) to (36x). It means that the image data transfer bandwidth from the processor (102) or (107) to the display element (203) is reduced by the foveal vision by a factor in the range (24x) to (36x). The thawing method of the present invention. If the FOV of the previous example near-eye display system (400) is greater than 20 °, for example at 40 °, the achieved compression gain of the peripheral region (407-412) is the image center region (402-406). Due to the large display FOV asymptotically approaching a factor 8 times higher than the compression gain achieved in, the peripheral image area constitutes the majority of the displayed FOV, and the foveated visual of the present invention. The decompression method can achieve higher synthetic compression gains (closer to factors higher than 40x) when the near-eye display system (200) has a FOV approaching that of HVS (HVSFOV may be greater than 100 °). Are known) .

  In another embodiment of the foveated visual decompression method of the present invention, the visual decompression transform provides a higher compression gain using different values of higher basis for image regions corresponding to the eye fovea 402. In order to achieve this embodiment, which provides the pafaovea 403-406 and perforvea 407-412 regions of the retina, the visual stretch transform (302) includes eye-gaze points (210) input from the eye and head tracking elements (210). Direction) (401) and then use the different values of the high criterion to create a transformed version for each image region that identifies the image region corresponding to the FOVea region (402). For example, the visual decompression transform (302) uses a (4x4) base to create a transformed version for the image region (402-406) and produces a (8x8) transformed image for use. And a visual decompression transform (302) for each image region by stitching the transformed images of the multiple regions prior to transmitting the transformed images of the multiple regions together. A quantizer (430), which sends embedded control data to the foveated image that identifies the basis order used, applies the basis coefficients to the appropriate truncation and quantization criteria, where each image region is The corresponding control data is transferred to the run length encoder (304). The run-length encoder (304) forwards to the run-length encoder (304) for transmission to the compressed display (203) using a higher order in the image area corresponding to the FOVea peripheral area, The affine transformation method of the present embodiment can achieve higher compression gain. In the above example, if the (4 × 4) base is used for the image regions (402-406) and ((8 × 8)), It is possible to achieve a compression gain asymptotically closer to a coefficient of (16x) than that achieved in the central image region (402-406) used for the peripheral regions (7-412). The foveated visual stretching method of this embodiment can achieve compound compression gains in the 32x-48x range for the previous example of a 20 ° display FOV, and in some cases Can reach 64x for a display FOV of 40 °.

  The described levels of compression gain that can be achieved by the foveated visual decompression method of the present invention translate directly into display 203 side processing and memory reduction, power consumption, volumetric morphology and Be able to translate directly into reduced costs. It should be noted that the processing and memory requirements of the visual stretch block (302) and the frustrated quantizer 430 block are that of the block of FIG. 4c. The latter is equivalent to that of a conventional image decompression element, except that it increases the image data bandwidth, thereby resulting in a considerable processing and memory requirement on the display 203 side with a proportional increase in power consumption. Cause an increase. Additionally, visual decompression (302) and foveated processing and memory requirements processing and memory requirements are provided. The quantizer 430 block of the figure is comparable to that of a prior art foretified rendering block, thus the near-eye display system (200) using the FOVed visual decompression method of the present invention. Incorporates a prior art forged rendering that requires significantly less processing and memory (thus reducing cost and power consumption) and uses conventional compression techniques (1) and It should also be noted that the Foveated Visual Decompression method of the present invention compared to the prior art near-eye display system of 1 (b) achieves its gain by matching the inherent capabilities of HVS; , HVS temporal integration and stepwise (or foveated) It provides spatial (angular) resolution (visual acuity), and the level of compression gain that can be achieved by the FOVed visual stretching method of the present invention is multi-view or multi-focus for processing. It is also important to keep in mind that when a system is needed, the near-eye display for displaying the light field is a significant amount, and the memory and interface bandwidth of such a system is It is necessary to display for a well-designed near-eye display system (which is directly proportional to the number of views or the number of focal planes (surface)), which gives a 3D perception level acceptable by the near-eye viewing viewer. It must be in the range of 6-12 views that need to be displayed in order to achieve.

  Foveated Dynamic Gamut—In another gamut embodiment, the visual stretch block is from the eye and head tracking elements to the pixel to which it next corresponds (with respect to the observer's gaze direction that maps to the macro). The center of the viewer's field of view is identified which identifies the center of the viewer's field of view and passes the spatial coordinates within the image frame that add that information to the image frame data to the quantizer block (303). Using spatial coordinates, the quantizer block produces a color sharpness profile for proportionally truncating the default 24 bits (8 bits) of a typical HVS (angle or orientation) applied color. The word length of the color coordinates of the image pixel (or macro) becomes the spatial coordinate of the center of the smaller sized (in bits) observer field of view. A word length corresponding to the position of each pixel (or macro) that, its frame. Pixel (or macro) color coordinate word length quantized coefficients are stored in a typical HVS (angular or directional) visual acuity profile (distribution) quantizer block (303) as a look-up table (LUT). Or macro), and a spatial distance from the center of the observer's visual field. Such an HVS color vision profile lut or generation function may be adjusted or biased by a given factor, depending on the taste of each particular viewer (based on typical viewer (angle or orientation)). It is possible. The gamut distribution corresponding to the HVS color sharpness profile is then quantized with the run length encoder (304) to quantize the quantized color values (added to the pixel (or macro)) and displayed for modulation. The described method of pixel '(or macro) transmission to element (203) is based on the visual acuity profile of the angle or directional color around the specified center of the viewer's field of view for each frame (t). Thus, reducing the word length is a compressed display that is the color FOVelation of the displayed image, which can lead to a factor of 2x-3x reduction in the size of the image frame data transferred to the display (203). , The display (203) displays the truncated color coordinates of the pixel '(or macro) it receives to modulate the image frame. To contact use. The term "foveated" as used within the context of this embodiment is meant to indicate that the display gamut is HVS color visual acuity profile (indicating that it is fitted to the distribution) from the eye center of the viewer. The visual compression gain of this embodiment, characterized in that it is arranged to move outwards towards the peripheral region of the retina of the observer's eye, causes the display to match the HVS color perception visual acuity distribution. Note that this is achieved by

  Near Eye Light Field Display-When sending different viewpoints of a scene image or video information to each eye, the viewer's HVS, which provides a near eye light field display, can fuse both images and communicate by the difference. The depth can be perceived. The parallax between the left and right images or video frames (3d perception) and what is known as stereoscopic depth perception. However, in conventional 3d displays, the depth perceived by the viewer, typically using the view for each eye, is the depth at which the viewer's eyes focus. May be different. This can lead to contention between the convergence provided to the viewer's HVS and the queue of containment depths (integrated containment contention, an effect known as VAC), which can lead to viewer headaches. There is a possibility of being connected. Eye distortion. By providing each of vt, the HVS of the viewer is characterized in that it is arranged to be visible to an observer with a corresponding perspective of the total light field period t in which the VAC can be eliminated. , At the same point in the light field, ie within the focusable light field, to allow for natural adaptation and focusing. The viewpoint of the light field presented to each of the observed eyes may be a sample (or slice) of the angle or depth of the light field. This approach is called a multi-view light field when the viewpoint presented to each viewer's eye is an angular sample of the light field, and a multi-focal plane light field when depth samples are used. Although their implementation details may vary, the approach of presenting the VAC-free light field to the viewer's HVS is a functionally equivalent representation of the light field. In either approach, the bandwidth of the visual data presented to the viewer's HVS is used to represent the field of view of the light field, which is proportional to the number of bright field samples (view or focal plane), and so on. Nana is a stereoscopic method that presents one view (or perspective) for each eye, which is much higher than before. The increase in visual data bandwidth can make it more difficult to realize near-eye displays that utilize the brightfield principle to eliminate VAC, which results in a corresponding increase in processing, memory. The following paragraphs realize a near-eye display that utilizes the brightfield principle to provide the viewer with a high quality visual experience while eliminating VAC and achieving compactness (streamlined lookup). Applying the described visual stretching method and other HVS vision matching methods to enable), after a practical near eye, a display system of either AR or VR is sought.

  Near-Eye Light Field Modulator-In one embodiment of the invention, visual information representative of a light field sample (view or focal plane) is presented or modulated by a near-eye display system) of an indicator (or light modulator). The groups of physical pixels are used to use the groups of right and left elements (203R,) 203L of the near-eye display (200) for the viewer's HVS, respectively. Here, the group multiplex physical (mxm) pixel of the light modulator element 203L is a group of light modulation elements 203R and (203L) called "(mm) modulation group" or "macro pixel". In the case of a multi-view light field near-eye display system implementation, where the macropixels used to modulate the light field samples (views or planes) called pixels (or m pixels) are called M pixels, each of the M pixels is The individual M pixels containing are used to be modulated (displayed). Multiple views of the light field are presented to the viewer's HVS, presented in the case of the multi-focal plane (planar)). To modulate (or display) multiple depth virtual image surfaces that represent the depth plane (sample) of the optical field HVS of the viewer that is used. The dimension of M pixels represents the total number of all-light field samples (expressed as (mxm)) and a near-eye display system is present in each of the viewer's eyes. In this embodiment, the HVS angular velocity that causes the characteristics of the light modulation elements 203R and (203L) of the optical (light emitting) near-eye light field display (200) to match the HVS angular velocity and FOV of the viewer is the viewer's. The viewer's HVS depth perception is highest in the foveal region of the eye and systematically decreases to the peripheral region 403-412 of the retina of the observer's eye, the highest in the fovea region (402) of the viewer's eye. Level and will systematically decrease to the peripheral region (403-412) of the retina of the viewer's eye. Therefore, by matching the viewer's HVS angular velocities, the light modulator elements 203L of the near-eye light field display of this embodiment are matched, as described in the following paragraphs, of the viewer's HVS. Adjust the angular depth vision.

  FIG. 5a illustrates an implementation of a light modulator (display) element 203L of a near-eye display system used to match the viewer's HVS angular velocity and FOV. In the figure, the M pixels of light modulator element 203L give the collimated luminous flux emitted from the M pixels that are emitting multicolor photonic microscale pixels (typically 5-10 microns in size). Of the optical modulator elements 203R and 203R are within the emission FOV of the optical modulator elements 203R and 203R. Also emitted from the associated M-pixel modulation group in which macro-optical elements (560) are provided (or evenly distributed) corresponding to each of the m pixels of the light-modulating element (203) shown in FIG. 5a. In order to achieve a given angular density of the directionally modulated light bundle, the light emitted from the associated M pixel onto the M pixel FOV is generated. The light flux emitted from each M pixel and modulated in the collimated direction is referred to herein as the "bright field angle". As shown in FIG. 5a, the dimension of M pixels moves away from the image modulation area corresponding to the optical aperture of light modulator elements 203R and 203L and the foveal center at the highest level in the optical center of the optical aperture. Thus, the HVS depth gradually decreases in proportion to the perceived visual acuity. As shown in FIG. 5a, in FIG. 5a, the angular range (FOV) of M pixels has a light modulator (which has the narrowest value at the optical center of the display) element (203R, 203L optical aperture). , Foveal center is configured to gradually increase in inverse proportion to the decrease in HVS angular velocity away from the image modulation region, which results in the highest angular density of the optical field anglet in the central region of the optical modulator. The (display) elements (203R, 203L) that have a value systematically decrease in the peripheral region.The light f / # of each of the light modulation elements (203R, 203L) shown in FIG. The optical aperture has the highest value in the central region of the optical aperture, and is located at a position that gradually decreases and is away from the image modulation region corresponding to the foveal center. In effect, in this embodiment, the light emitted from the light modulator element 203L has the highest resolution available in the image area targeted for the viewer's HVS visual acuity in the orbital area 402 of the observer's eye. In accordance with the HVS visual acuity distribution at the time of creating, and the systematic decrease in the range of movement of the pupil of the eye of the observer in the peripheral region 403-412 of the retina of the observer's eye from the far field (It should be noted to match up to (−7 °)), as shown in FIG. 5a, the central region of the highest resolution (central ± 5 ° FOV region of FIG. (5)) is The positions of all possible eye FOVeaFOV regions (402) within the observer's eye movement are made wide enough to accommodate the viewer's near-field to far-field. After describing a method for implementing a light field modulator that is optically matched to HVS vision, the next paragraph describes a method of using HVS optically matched light modulator elements (203R, 203L). To do. It can be used in conjunction with the previously described focused visual stretching method to achieve a near-eye light field display that uses either the multi-view or multi-focal plane light field sampling methods described above. Ah

  Multi-view light field-FIG. 5b shows at a high level the coupling between the optical element (206) and the light modulator (display) element (203R,) 203L of the previous embodiment. In the near-eye display system (200), the optical element 206 shows the exemplification of FIG. 5b. The image modulated by the display elements (203R,) 203L is appropriately magnified by the optical element (206) into the eyes (580) of the observer. The relayed optical element (206) can be implemented using reflector and beam splitter optics assemblies, free-form optical wedges or waveguide optics. Although these optical elements 206 have different design options, the common design criteria for them is to view the optical modulator (enlarge and relay the optical output of the display) elements (203R,) 203L. Selected m pixel (mxm) dimensional design criteria to be transmitted to the human eye (580) and the effective optical magnification from the optical modulator elements (203R, 203L) and the macro optical elements (555), (560). Provide spatial acuity for virtual images formed (modulated) with mVS pixel spot sizes located in the central optical regions of the light modulator elements (203R, 203L) at HVS (average) minimum viewing distances, respectively. Near-field of near-eye display system (200) covering (402-404) of Figure 4a. For example, for a near-eye display system with a minimum viewing distance of 30 cm and an HVS spatial visual acuity at that distance of about 40 microns, of the m pixel in the central light center region of the light modulator element (203R,) 203L. The pitch is also 40 μm, the pitch of m pixels of the light modulator elements (203R, 203L) is 10 μm, the dimension of M pixels is (4 × 4) M pixels, the central region of the eyes of the observer (FIG. For each of (402-404) of (4a), allows the near-eye optical field display system (200) to modulate up to 4x4 = 16 views, in this example M pixels, as shown in Figure 5a. Is characterized in that it is configured to systematically present a (1x1) number of views that gradually decreases to (3x3), (2x2). It includes the peripheral region (405-412) of the observer's FOV, and thus the light modulator elements (203R, 203L) of this embodiment have the viewer's foveal region (402-404 on FIG. 4a). By modulating more views to produce a systematically smaller number of views into the peripheral region (405-412) of the viewer's FOV (consistent with the viewer's HVS angular velocity and depth perceptual aspects). Is configured. This effect is a form of visual compression. Because the highest number of views needed to provide the viewer with the highest depth cues is modulated by the light modulator elements and 203L within the viewer's foveal region (402-404 in the figure). A), as indicated by the observer's line-of-sight direction, and the depth of focus sensed by the eye and head tracking elements (210), as indicated by the observer's line-of-sight direction and depth of focus, and systematic. A very small number of views are modulated onto the peripheral area 405-412 of the viewer's FOV in proportion to the typical visual angle distribution of HVS. Thus, in the previous example, 16 views would be modulated by the display element 203L into the central region of the viewer's eye (on 402-404 in the figure). The bandwidth of the image input (301), which is a width of about 2 °, with a small number of views modulated into the peripheral area (405-412) of the observer's FOV, is set to the central area of the viewer's eye, FIG. It is configured to change in proportion to the total angular width of the near-eye display (200FOV) (reduced so as to be approximately proportional to the angular width of (402-404)). That is, for example, if the near-eye optical field display 200FOV is 20 ° wide, 16 views are modulated into an angular region of 2 ° in its center and an average of 4 views in its peripheral region, about 5 views. The effective bandwidth of is sufficient in such cases, which is equal to the compression gain of a factor of (3x). Of course, with the visual compression method of this embodiment, a higher compression gain will be achieved if the near-eye display 200FOV is wider than the 20 ° envisioned in the illustrative example.

  Multiview Light Field Depth Foveated Visual Thawing-HVS angle (perception) visual acuity from the center towards the peripheral area decreases systematically from near field (-30 cm) to observer far field It is configured to move toward (-300 cm). Therefore, HVS will require a higher number of views for near field depth perception than far field depth perception. In addition, the HVS depth perceptual visual acuity is at its highest level near that point in that it can be stored in a particular location with focus on the viewer's eyes, with no depth or angular deviation from that point. It is systematically reduced by either. This contributes to the visual information in the vicinity of the point where the viewer's eyes are in focus, and can contribute to maximizing the depth feeling. The number of such views systematically decreases as the viewer's eyes change from the near field to the far field of the viewer. This attribute of HVS depth perception presents yet another visual compression opportunity that can be exploited by a combination of (foveated) and multi-view light modulator elements (203R, 203L) of FIG. In one embodiment incorporated within the near-eye light field display system 200 including the described foveated visual stretching method, the multi-view light modulator elements and 203R, 203L of FIG. Identify (or identify) using the sensed points of the viewer's focus provided by the eye and head tracking elements (210), including both foveated visual decompression methods) , Contributes the most visual information in the vicinity of the point where the observer's eyes are in focus, and then the described visual decompression method is applied Eye viewer with respect to the contribution to the visual information in the vicinity of being focused, multiview light modulator elements 203R in FIG. 5a, the 203L, compressing the bright field views that are modulated to the viewer proportionately. [Means for Solving the Problems] Therefore, according to the method of the present invention, in the embodiment, the light field contributes the most visual information in the vicinity of the point where the viewer's eye is t. The highest number of modulation basis coefficients collected by the 5a polyhedral light modulators (203R, 203L) to achieve one t are used to obtain the highest visual perception with the least truncation of their word length representations. While allowing, at a higher word length truncation than the brightfield view, which has less contribution near the point where the viewer's eyes are present, a proportionally smaller number of modulation basis coefficients are used to provide a wider angle. Modulated by the multi-view light modulator elements (203R, 203L) of FIG. 5a using fewer light field modulated views spaced by pitch. The net effect of the method of the present embodiment uses the visual information of the surrounding area (front, back, and side areas) which is a three-dimensional FOVelatedvisualdecompression action reflecting the visual information near the point where the viewer's eyes are in focus. However, at the viewer's focus, it will be modulated with the highest fidelity that matches the HVS perceptual visual acuity) is proportional to the HVS at a point away from where the viewer's eyes are focused. Is modulated with a fidelity level that corresponds to a relatively small perceptual acuity. The combined methods of this embodiment are collectively referred to as "multi-view light field depth foveated visual decompression." It should be noted that the term "fovea" used within the context of the present embodiment means that the display resolution indicates that the HVS depth perception visual acuity profile (fit to the distribution) is visible. It is characterized in that it is configured to move outward from the center of the human eye toward the peripheral region of the retina of the observer eye.

  Note further that in the previous embodiment, the higher number of views are modulated by the display element and 203L into the central region of the viewer's eye (on 402-404 in FIG. 4a). As shown by the tracking element (210), the display element (203R,) 203L modulates the highest possible number of views over an angular region extending across the angular distance between the near and far fields of the observer. Can be done, which is about 7 ° in total. Nevertheless, when the foveated visual decompression method of the previous embodiment is applied, the present invention has been made to solve the above-mentioned problems. This will truncate and quantize the modulated basis coefficients to match, thus, as explained previously, the compression gain of the foveated visual stretch and the foveated of FIG. 5a. The multi-view light modulator element (203R, 203L) can be combined. That is, the foveated visual stretch that achieves a medium compression gain factor of (32x) achieves the compression gain factor of (3x). The foveated multi-view light modulator of FIG. When combined with the elements (203R, 203L), the composite compression that can be achieved by the near-eye multi-view light field display system compared to the same as in the previous example achieves a comparable viewing experience. Compared with display systems, it reaches a compression gain factor of (96x) and provides 16 views with near-eye optical field display capability of the eye.

Multifocal Plane (Surface) Light Field-Figure 6a shows an embodiment of applying the inventive visual stretching method in the context of a multifocal plane (surface) near-eye light field display. As shown in FIG. A, in the present embodiment, the m pixels and m pixels of the optical field modulator are the m pixels and m pixels of the optical field modulator (203R, 203L) are modulated in the collimated direction. Embodiment of a near-eye light field display (200) that is designed to produce a bundle of rays (or a bright field angle) that are collectively angled (61) or 610L. In, a near-eye light field display (200) is provided with left and right light field modulators (203R, 203L) comprising a plurality of m pixels and 2t m pixels designed to produce left and right eyes of an observer. Configured to produce a left and right light field anglet pair (660OR) 610L addressing a corresponding point on the retina (580R, 580L) (The retinal corresponding point is the point on the retina of the opposing eye of the observer whose perceptual output is perceived by the observer's visual cortex as a single point in depth.) The left and right optical field angle pairs (61OR, 610L) generated by the containers (203R, 203L) are called "visually corresponding", and the optical field angle pair 61OR, 610L address addresses t. Each set of corresponding points on the retina of the left and right eyes (580R, 580L) of the observer, sometimes referred to as "visually corresponding," is set to the point in the FOV of the near-eye light field display (200). Generated by the right and left light field modulators (20R, 203L) and by the optical element (206) the observer's eye ( 80R), a “visually corresponding” light field angle pair (660OR) 610L (580L), is perceived by the observer's visual cortex as a virtual point (vpos) of light twinned. (620) is in the light field modulated by the near-eye light field display system (200), and the binocular perceptual aspect of the viewer's HVS is relayed onto the observer's eye (580R, 580L) retina. A virtual point of the perceived depth (VPoL) (620) that transforms the captured light flux image into a light flux into a single viewpoint by an optical element (206), corresponding to the eyes of the viewer (580R, 580L). In this embodiment, the near-eye optical field display (200) modulates (or generates) a virtual point (vpos) (6). 20) is a "bright field angle 61or, 610L" that allows the "visually corresponding" pairs of t to be simultaneously perceived by the viewer in the display FOV and the left and right light field modulators. The position of the virtual point of the light (vpos) (620) perceived by the observer in the field of view having a "bright field angle 61or, 610L" by 203R, (203L) is , X, y) r and x, the y) l space (coordinate) position is the left and right light field modulations that have produced a “visually” pair in the left and right light field modulators 203R, 203L. Indicates the position of m pixel and / or m pixel in the container 203R, 203L
Corresponding "light. Field. Angle 61or and 610L. X, y) by addressing r and x, y) left and right optical field modulator (20R,) 203L m pixel and / or m pixel l spatial position. The near-eye optical field display (200) can modulate (generate) a virtual point of light (vpos)) (620) is an arbitrary depth within the FOV of the near-eye optical field display (200). According to this method of VPoL 620 modulation, which is commonly perceived by observers in, a near-eye light field display is capable of modulating three dimensions (3D) "visually corresponding" light field. ) Or (61) by modulating the pair of viewers in the display FOV Possible Light.Field. The term "viewer focusable" configured to perform (61) or's respectively by right and left light field modulators (203R,) 203L with selectable content. Is used in this context to mean that the viewer of a near-eye light field display can focus on an object or content)) is placed in a modulated light field. Is a key feature of near-eye optical field displays that contribute significantly to reducing the aforementioned VAC problems experienced by conventional 3d displays.

  Due to the inherent ability of HVS depth perceptual vision, all possible virtual point light (VPoL) addressing (620) can be placed in the field of view, no near-eye light field display (200) required Is. The reason is binoculars. In the corresponding area (an HVS-based perceptual aspect in which binocular depth perception is achieved by looking at an object at a given motion distance (or position) from the viewer's eye, forming images at points) An image processing device characterized by being present. The trajectories of all such positions deviate (or move) from all such positions, and the eyes of the observer describe the angular distribution of HVS visual acuity, known as the horopter plane, as its binocular depth perception. The combination produces a depth region surrounding the horopter surface, known as the fusion area or volume of the panum), even if the object perceived by the observer is not actually in the horopter plane. Perception of eye depth is achieved. [Means for Solving the Problem] In order to solve such a problem, the present invention relates to a related fusion area of the panum, which causes a light field to be determined by the approximate size of the fusion area of the panum. We propose a method for sampling into separate discrete surface sets and, of course, guarantee continuity of binocular depth perception within the volume between the light field sampling planes, with some overlap. Empirical measurements (Hoffman, M; gishck, aR akley, K. and bank, Ms. impede visual performance, visual fatigue, Journal of Vision (2008), (8), ( 33), demonstrating that continuity of binocular depth perception can be achieved when having multiple 2d light modulating surfaces separated by a diopter of approximately 0.6 degrees (causing (1-30)). (D) It exists within the visual field of the viewer. The set of horopter planes 6 in the observer's FOV separated by (0), therefore d is such that multiple horopters within the range of the viewer's HVS are sufficient to achieve binocular perception. It provides a volume that spans the fusion areas of the faces and their associated panum. In the present specification, horopter planes separated by a distance necessary to achieve continuity of viewer's binocular depth perception are provided to achieve continuity within an FOV extending from the viewer's near to far field. .

  In this embodiment, the described method of sampling the near-eye light field to canonical (meaning enough to achieve continuous stereoscopic binocular depth perception) is provided) (0.6D). Achieved using the described virtual points of the light (VPoL) 620 modulation method of the near-eye optical field display (200) described in the set of discrete horopter planes (separation distance of horopter planes) x), y) r and x 1) spatial positions of m pixels and / or m pixels, achieved by defining a set of r and x, in the right and left optical field modulators 203R, 203L, respectively. Both viewers of multiplicity of light (vpos) (620) in a selected reference set of horopter planes in a display system 200FOV that produces a "visually" set. A "corresponding" bright viewing angle "that provides the perception. Using the described virtual points of the light (vpos) 620 modulation method, a method is provided to modulate the canonical set of horopter planes, thereby providing a near-eye light field. The display (200) is capable of perceptually addressing a viewer's entire extra-ocular light field, thus the method of this embodiment effectively achieves optical field compression gain, and this optical field compression is achieved. The gain is a considerable compression gain expected to exceed (100x), selecting the selected horopter modulation plane for the size of the total optical field (vpos) addressable by the near-eye optical field display (200). (Note that such compression gain is achieved by a virtual point of light (vpos)). (620) and the binocular perception of HVS and the angular velocity are matched with each other, the modulation capability of the near-eye optical field display (200) is adjusted.

FIG. 6b illustrates the near-eye optical field horopter sampling and modulation method of the previous embodiment. FIG. 6b shows a top view of the bright field honing surface (615). (622, 625), (630), (635), (640) are systematic from near field (−30 cm) to far field. The viewer's (-300 cm) relative to the position of the observer's eye (610). As shown in FIG. 6b, the first bright-field horopter plane (615) is at the near-field distance of the observer located 3D above the observer's eyes and the remaining (5) bright-field horopter planes ( 618) is away from the observer's eyes, (63-640) is located at (73a,) 2D, a continuous (0.6D) distance from the viewer's eyes (1). From the eyes of the viewer, (53D), (0.93D), and (33D) are output, respectively. The (six) brightfield honing surfaces (615), (618), (625), (630), (635), (640) each consist of a number of vpol 620 modulated in density (or resolution). Corresponding to the depth of the HVS and the acute angle of the angle, for example, the density (spot size) of the modulated vpol 620 is the density (spot size) in the first light field in the horopter plane. (615) is 40 μm to match the HVS spatial visual acuity at that distance, and continuously increases in the remaining (5) bright field horopter planes (618), (625), (630), ( 635) and (640) are arranged so as to match the distribution of HVS space and angular velocity. Six bright field horopter planes 615, 618, 625, 630, 635. The multiple vpols 620 that make up each of the 640 are the "visually corresponding" light field angle pairs (61or) (610L) generated by their corresponding multiplicities that are modulated (generated). The (x, y) r and (x, y) l spatial positions produced by the M pixels and / or the defined set of M pixels located in each of the left and right sides of the near-eye light field display (200), respectively. It is a spatial position in the optical field modulators 203R and 203L. Modulates (generates) spatial positions (x), y) R, x, y) l in the left and right optical field modulators 203R and 203L (6). Each of the bright-field angle planes (615), (618), (625), (630), (635), (640) of the visual extension angle conversion plane ( 02) is calculated in advance by, maintained, it is converted into the address
Embedded or external based on their corresponding vpol 620, including each of the (six) brightfield honing surfaces (615), (618), (625), (630), optical field image data (301). Received as input from either of the processors (102) or (107).

  Visual Defrosting with Depth Foveation in Multifocal Plane Light Field-Left and Right Light Field Modulators 203R, 203L of Near-Eye Light Field Display System (200) are (6) Light Field Hopter Surfaces (615) , (620), (625), (630), (635), (640) are simultaneously included, which means that at a particular moment, the viewer's eyes are at a certain distance. Focused on, and as described above, HVS depth perception visual acuity is at its highest value in the vicinity of that point, and should be systematic in either depth or angular deviation from that point. Decrease to. Therefore, in the present embodiment, the multi-focal plane display system (200) of the present invention uses the multi-focal plane optical field modulation method of the present invention (six) bright-field angle planes (615), (618), (6). (635), (635) with (630), (640) are vmod 620 density (resolution) that is simultaneously modulated but is consistent with HVS vision at the viewer point of focus. In addition, embodiments incorporated within the near-eye display system 200 include both of the described methods of modulating the near-eye optical field using vpol 620, which modulates the canonical horopter plane (615) (625 ), (630), (635), (640) are focused on the observer's eyes as shown in FIG. 6b and using the foveated visual stretching method described in the previous embodiment. The horopter surface that contributes the most visual information in the vicinity of the point is applied, and then the described foveated visual decompression method is applied to modulate the (six) light fields vpol620. Proportional to the contribution to the visual information in the vicinity of the observer's eye, which compresses proportionally to the horopter planes (615), (620), (625), (630), 635), to form a (640). In this embodiment, a horopter surface less than (0.6D) less than where the observer's eyes are in focus (distance of motion). This criterion includes (625), (630), (635), (640) identifying at least (two) of the canonical brightfield honing planes (615), with the focus of the viewer on these. A horopter surface is identified if it is not directly on one of the surfaces. This is because the binocular fusion area of the observer's HVS fills the area of (0.6D) between the canonical brightfield horopter planes as described above. In this embodiment, which ensures that the optical depth falls within the binocular fusion zone of at least one of the selected (identified) optical field horopter planes, identification is performed using the described selection criteria. The resulting horopter plane contributes to the most visual information in the vicinity of the point where the viewer's eye is in focus and accommodates the multi-focal plane light modulators (displays) of the present invention. Modulated horopter surfaces to obtain the best visual perception using vpol620 densities consistent with HVS visual acuity at the perceived depth of these surfaces, and using the highest number of modulation basis coefficients at tt. Multiple to achieve The focal plane light modulator (display) scales proportionally at higher word-length truncation, which allows the rest of the horopter plane to be modulated with less contribution near the point where the observer's eye is in focus. A smaller number of modulation basis coefficients are used to form fewer elements (203R, 203L) in FIG. 6a, using less vpol 620 separated by a wider angular pitch. The net effect of the method of this embodiment is the HVS at the focus position, which is a three-dimensional fossilized visual stretching motion in which the visual information near the point where the viewer's eyes are in focus is modulated with the highest fidelity. Visual information in the surrounding area, consistent with the perceptual visual acuity (modulated at a fidelity level consistent with the proportionally smaller perceptual visual acuity of the HVS at points anterior, posterior, or lateral) observation Focus on the eyes of the person. The combined method of this embodiment is collectively referred to as multifocal plane light field depth foveated visual decompression. It should be noted that the term "foveated" as used within the context of this embodiment is meant to indicate that the display resolution is fitted to the HVS depth perceptual visual acuity profile (distribution). It is characterized in that it is configured to move outward from the center of the eye of the observer toward the peripheral region of the retina of the observer's eye.

  Note that in the previous embodiment, the higher density of vpol 620 is modulated by the illustrated display element 203L into the central region of the viewer's eye (402-404 in FIG. 4a). As shown at (210), the display element (203R,) 203L of FIG. 6a, which has the highest possible vpol 620 over an angular region extending over the angular distance between the near and far fields of the observer. It is possible to modulate the density, which is about 7 ° in total. Nevertheless, as explained above, when the depth foveated visual decompression method of the previous embodiment is applied, it adjusts the modulation basis coefficients to match the HVS angle and depth perceptual visual acuity. Effectively combining the compression gains that would be truncated and quantized, decompressed and foveated multi-focus planes light modulators for the display elements (203R, 203L) of FIG. 6a (display ). That is, if a foveated visual stretch that achieves a (32x) medium compression gain factor is combined with the foveated multifocal surface light modulator elements 203R and 203L of Figure 6a: Similar to the previous example, while achieving a compression gain factor of about 3x (in selecting only the most (2) of the (6) canonical Hotel surfaces), all 6 positive The composite compression that can be achieved by the near-eye light field display system (200) achieves a comparable viewing experience (6), as compared to quasi-hotersurface centered vpol 620 density). A focal plane function reaching a gain factor of (96x) compared to a near-eye display system using a near-eye optical field display with.

  FIG. 7 illustrates the generation of content for the multi-focal plane near-eye light field display of FIG. 6a. In this illustrative example, a scene is captured by a camera (701) in (3) depth planes, namely near, mid, and far (3) depth planes. Note that the deeper plane captured by the camera provides better depth perception for the viewer on the illustrated multi-focal-plane light field near-eye display 200. Preferably, the number of capture depth planes should be proportional to the number that the number of near-eye displays in the focal plane of Figure 6a can be modulated, in the case of the previous embodiment (six) positive. See (618), (625), (630), (635) and 640 of FIG. 6b, which are quasi-horopter planes (615). This, however, the example uses the capture plane of to illustrate a further aspect of the present invention, one of ordinary skill in the art would use the methods described herein to provide multi-focus planar near-eye imaging (capture and display). (3) is provided in this exemplary example of providing a system using three or more captured depth planes of an exemplary embodiment of the invention. Object is placed in the content scene, object (702) is close to the capture camera, and other (two) objects (703) and (704) are far from the camera. In a multi-focal plane imaging system, a depth layer (where the brightness of the objects is adjusted according to their position relative to the capture) is required. In the illustrative example of FIG. 7, this is accomplished by depth filtering, as shown by filtering blocks (705), (706), (707), and the intensity of the image scene object, At least one of the brightnesses of the image contents is determined so that the brightness corresponds to the depth value. For example, the closest object (702) is shown in full brightness within that particular layer 708, which is fully contained in the first depth layer, but the other (two) layers (706). ), The middle object (703) is completely removed from (707), the (two) depth layers (located between the middle and far) render the full brightness of the object (703). In order to do, the full brightness is divided between the (two) layers (706), (707), the perceived brightness of the object is the sum of all layers (71) Is perceived at full brightness by the observer's eyes as a weighted sum of the luminance contributions from both depth planes (706), (707). In order to achieve a 3d perception of the scene, depth layers, and 710 Each of which is visible to the viewer at that time Corresponding depth, brightness is adjusted, arouse depth cue viewers effectively, in order to enable focusing by the viewer to the displayed content, consistent with the depth of the scene objects. The viewer will see a combination of all layers, resulting in a reconstructed stereo image (71) of this exemplary example, which provides the proper focus cues for the viewer's HVS. The image content of the capture planes, together with their relative depth information, as described above, is the color and brightness of their image content (see Rendered for Multi-Focal Plane of FIG. 6a). The end result of this captured image rendering process of projecting onto the multi-focal plane is the content color and brightness of the input image (301), "plural" "visually" color and brightness data. To a corresponding data set that identifies each of the M pixels and / or the corresponding "Layers" generated by each set of M pixels (x, y) r. Doo field. angle pair 61or and 610L "and the (x, y) l spatial position is the spatial position within the left and right optical field modulators 203R, 203L of the near-eye optical field display (200), respectively, for these color and luminance data sets. Respectively by the left and right light field modulators (20R,) 203L of the near-eye light field display (200), the viewer perceives the rendered 3d image input as a set of modulated vpos 620 and tit. It should be noted that in the previous illustrated example where any of the displayed 3d objects (702), (703), (704) could be focused in the scene, only the capture plane of was used. The near-eye optical field display (200) of the present invention is The input image data (301), which is still rendered in this case using the described method of the present embodiment, uses the described vpol620 modulation scheme and its near-eye light field display capability. To display the input image content, the horopter surfaces that are input on the six canonical horopter surfaces of Fig. 6b are identified. Because it fills the (0.6D) region between the quasi-brightfield horopter planes, this criterion is that the optical depth of the observer's focal region is that of the selected (identified) optical field horopter plane. In this embodiment, which ensures that it falls within at least one of the binocular fusion areas, the horopter plane identified using the described selection criteria is aligned with the eyes of the viewer. The multi-focal plane light modulators (displays) of the present invention, which contribute to the most visual information in the vicinity of the contained point, modulate these identified horopter surfaces to provide a sensed sense of these surfaces. An observer with a multi-focal-plane light modulator (display) that achieves the best visual perception using vpol620 density that matches HVS visual acuity in depth and using the highest number of modulation basis coefficients in tt. In the higher word-length truncation that causes the rest of the horopter plane to be modulated with less contribution near the point where the eye is focused, using a proportionally smaller number of modulation basis coefficients, Fewer vpols 620 separated by a wider angular pitch are used to form the elements (203R, 203L) of Figure (6) The net effect of the method of this embodiment is viewing. Information in the surrounding area that matches the HVS perceptual visual acuity at the focus position, which is a three-dimensional fossilized visual stretching motion in which the visual information near the point where the eye is focused is modulated with the highest fidelity Focuses on the observer's eye (modulated with a fidelity level corresponding to the proportionally small perceptual visual acuity of the HVS at front, back, or at a point distant from the side). However, the example uses a capture plane of to illustrate a further aspect of the present invention. Those skilled in the art will use the methods described herein to describe multi-focal plane near-eye imaging (meaning capture and display). In this illustrative example, which provides a system using three or more captured depth planes of the illustrative embodiment of the present invention, (three) objects Are placed in the content scene, the object (702) is close to the capture camera, and the other (two) objects (703) and (704) are far from the camera. In a multi-focal plane imaging system, a depth layer (where the brightness of the objects is adjusted according to their position relative to the capture) is required. In the illustrative example of FIG. 7, this is accomplished by depth filtering, as shown by filtering blocks (705), (706), (707), and the intensity of the image scene object, At least one of the brightnesses of the image contents is determined so that the brightness corresponds to the depth value. For example, the closest object (702) is shown in full brightness within that particular layer 708, which is fully contained in the first depth layer, but the other (two) layers (706). ), The middle object (703) is completely removed from (707), the (two) depth layers (located between the middle and far) render the full brightness of the object (703). In order to do, the full brightness is divided between the (two) layers (706), (707), the perceived brightness of the object is the sum of all layers (71) To achieve a 3d perception of the scene as perceived by the observer's eyes at full brightness, as a weighted sum of the luminance contributions from both depth planes (706), (707), and depth layers, and 710. Each of which will be displayed to the viewer at that timeThe corresponding depth corresponds to the depth of the scene object so that the adjusted brightness effectively evokes the viewer's depth cue and allows the displayed content to be focused by the viewer. The viewer will see a combination of all layers, resulting in a reconstructed stereo image (71) of this exemplary example, which provides the proper focus cues for the viewer's HVS. The image content of the capture planes, as described above, together with their relative depth information, are rendered to be delivered (mapped) the color and brightness of their image content, as shown in the multi-focus of figure (6). The end result of this captured image rendering process is to project the content color and brightness of the input image (301) into "multiple" visually "colors and brightness. Is mapped to a data set that specifies a corresponding set of M pixels and / or a corresponding set of M pixels (x, y) r. Itofield. The angle pairs 61or and 610L "and the (x, y) l spatial position are the spatial positions within the left and right light field modulators 203R, 203L of the near-eye light field display (200), respectively. The luminance data set is modulated by the left and right light field modulators (20R,) 203L of the near-eye light field display (200), respectively, and the viewer views the rendered 3d image input content as a set of modulated vpos 620 and tit. In the previous example, which could focus on any of the 3d objects (702), (703), (704) displayed in the scene, the only capture plane of was used. It should be noted that the near-eye optical field display of the present invention The ray (200) is still rendered in this case using the described method of this embodiment, the input image data (301) is the near-eye light field using the described vpol620 modulation scheme. Inputs are made on the six canonical horopter planes of FIG. 6b to display the input image content using the display function.

  The multi-focus depth filtering process shown in FIG. 7 is a process of assigning (mapping) the brightness of the input image scene content, using an objective lens on the display device (200) depending on the relevant input image depth information. In one embodiment of the present invention for converting a multi-focal plane to a multi-focal plane to create a perceived depth cue appropriate for a viewer's HVS, the multi-focal plane near-eye light field display (200) of the present invention comprises: In the case of the previous embodiment, where local depth filtering can be performed to generate all depth layers used by the near-eye light field display (200) of FIG. 6a with t, The display FOV, which is within the range characterized in that the (six) canonical horopter planes are located on t, is As shown in FIG. 6b and FIG. 8, which is characterized in that the field of view is from the near field to the far field of it, the multi-focus plane depth filtering method (825) of the present embodiment uses the layer splitter (802) as an image. The input (301) and its associated depth map (801) showing processing 2t produces an image depth plane or layer corresponding to the capture depth plane. The content of each generated layer is then depth filtered (803) for display by mapping the input image (301) and its associated input depth map (602) to a multifocal plane image. The image rendering block then uses the generated multi-focal plane image to generate a plurality of "visually corresponding" light field angletlets to be generated by each set of M pixels and / or M pixels x. Display the multi-focal plane vpol 620, which produces a pair and a color and intensity value of 610 l, where the y) r and (x, y) l spatial positions are the spatial positions within the left and right optical field modulators 203R, 203L, respectively. At least one of the near-eye light field displays (200) for modulating the viewer's.

  In another embodiment, the horopter planes (615), (620), (625), (630), (635) of the T near-eye light field display (200) in which the display image for the canonical light field is generated. , (640) the previous embodiment describes a captured scene content (generated from an input image (301) composed of a set of compressed reference element images or holographic elements (hogel)). Reference) In this embodiment, the elemental images or hogels captured by the brightfield camera of the scene are first processed to identify the largest or fully contributing minimum number of captured elemental images or hogels that identify the processed t. A subset of the canonical brightfield horopter multifocal planes (615), (620), (representing (specified) image content) 25), (630), (635), (640. The depth to identify the) source of the scene a subset of the identified element image or hogel, in this specification, referred to as reference Hogeru. Light. field. Data for the total number of elemental images or hogels captured by the camera. Determine the data size of the identified reference hogel, including the image content of the standard multifocal plane (615), to be compared with the size (618), (625), (630), (635), (640). ) Represents the compression gain that is inversely proportional to the data size of the identified subset of reference ho-gels divided by the total number of captured elemental images or ho-gels. Is. Thus, in this embodiment, the captured light field data set is compressed into a data set representing a discrete set of multi-focal planes of the near-eye light field display (200) and, thus, the canonical light. Including (618), (625), (630), (635), (640) identified by the method of the previous embodiment, where a compression gain reflecting the field horopter multifocal plane (615) is realized. Is a compressed representation of the optical field that achieves compression gain by matching the viewer's HVS depth perception aspects.

  Compressed Rendering-In another embodiment, as shown in FIG. 9, a "compressed rendering" is performed directly on the received image input (805) containing the compressed image (US Patent Application Publication No. 2015). No. 020176) is executed. In order to extract the image to be displayed by the multi-focal plane near-eye light field display (200), the bright field dataset of the reference hogel of the previous embodiment and the light at the canonical light field horopter multifocal plane (615). And left and right light field modulators (203R,) 203L for modulating a field image, the figure shows the compressed rendering process of this embodiment, and the compressed rendering process of this embodiment is , Input light field. The data processes a compressed light field dataset consisting of a reference hogel to produce inputs to a multi-focal plane near-eye light field display (200) and a left light field modulator (203R, 203L). In the compressed rendering process (806) of FIG. 9, the received compressed input image (805) is the reference hogel's light at time t. field. data. The previous embodiment including the set is first rendered to extract the light field images at the canonical light field horopter multifocal planes (615), (620), (625) (630), (635), ( 640.) and in a first step (810) of the compressed rendering process (806), the reference hogel images produce an image with their associated depth and texture data, including the light input (805), Is used to synthesize the color and luminance values of the near-eye optical field vpol consisting of each of the canonical field horopter multifocal planes (615), (618), (625), (630), (635), ( 640) The reference hogel is a canonical optical field horopter multifocal plane (615), (618), (625), (630), (635), ( The vpos compositing process (810), which was chosen a priori based on the depth information of 640), requires minimal processing throughput and memory, and uses near-eye light from the compressed reference hogel input data (805). Extracting field vpol color and intensity values In addition, the viewer's gaze direction and depth of focus sensed by the eye and head tracking elements are used by the vpos synthesis process to sense the gaze direction and the viewer's focus. Render vpos values based on the viewer's HVS visual acuity distribution profile versus depth. The value associated with each of the combined near-eye light fields vpol will be a visually identified pair. The corresponding angle directions and their (x, y) r and (x, y) l spatial position coordinates are respectively located within the left and right optical field modulators 203, respectively, and are associated with the visually corresponding angle pairs. Are mapped (transformed) to the right and left optical field modulators 203R, 203L, located on (x, y) r and (x, y) l spatial position coordinates, And depth-tracked visual compression blocks, respectively determined by the angle synthesis process (81515) in response to the observer's gaze direction sensed by the eye tracking element (210) and the eye tracking element (210) described in the previous embodiment. The method is used to compress the generated color and luminance values, and the (X), y) and (x, y) l spatial position coordinates are based on the viewer's HVS visual acuity distribution. Left and right optical field modulator 203R, a respective spatial position coordinates of the 203L. In essence, this embodiment was input to (3) the compression gain of the previous embodiment, namely (1) a set of minimum reference hogels that completely compose the canonical optical field multifocal plane. Determining the compression-related gain of the optical field data, (2) converting the compression-related gain of the entire optical field into a set of vpols each containing a canonical optical field multiple focal plane, and (2) (3) The first of these compression gains, which determines the gain associated with the depth FOVelation of the modulated vpos to match the viewer's HVS angle, color, and depth vision, is substantially First, it reduces the interface bandwidth of the near-eye display system, and the second of these compression gains produces computation, which substantially reduces processing. A third of these compression gains substantially reduces the interface bandwidth of the near-eye display light field modulator (203R,) 203L, and the effect of these compression gains is now seen in prior art display systems. Note that this is further enhanced by the compressed display capability of the near-eye display light field modulator 203L, which allows direct display of the compressed input without the need to decompress the first compressed input as is done. I want to be done.

  The previous description of embodiments presents an image compression method for a near-eye display system that reduces input bandwidth and system processing resources. Combined using a compressed input display that provides higher order basis modulation, dynamic color gamut, optical field depth sampling, and image data word length reduction and quantization for matching angles in the human visual system. Color and depth vision enable high fidelity visual experience in near-eye display systems suitable for mobile applications with substantially reduced input interface bandwidth and processing resources

  Those skilled in the art will readily appreciate that various modifications and changes can be applied to the embodiments of the present invention without departing from the scope of the appended claims and the scope defined by the claims. Ah It should be understood that the above-described embodiments of the present invention are exemplary only, and that the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. For example, various possible combinations of the disclosed embodiments can be used together to achieve additional compression gain in near-eye display designs not specifically mentioned in the previous exemplary embodiments. Accordingly, the disclosed embodiments are not to be considered in any sense as limiting, individually or in any possible combination, the scope of the invention is not limited to the preceding description, but to the accompanying description. All modifications that are indicated by the claims and fall within the meaning and range of equivalents thereof are intended to be embraced therein.

  The present invention relates generally to compression methods for image generation systems, and more particularly to images and data for head-mounted or near-eye display systems (collectively referred to herein as near-eye display systems). Regarding the compression method.

  Near-eye display devices have received widespread attention in recent years. Near-eye display devices are not new, and many prototypes and commercial products can be traced back to the 1960s, but due to advances in network computing, embedded computing, display technology, and optical component design, Renewed interest in devices. Near-eye display systems are typically coupled with a (embedded or external) processor, tracking sensors for data acquisition, display devices, and necessary optics. The processor typically serves to process the data obtained from the sensors to produce data that can be displayed as a virtual image within the visual field of one or both eyes of the user. This data can range from simple warning messages or 2D info charts to complex floating animated 3D objects.

  Two types of near-eye displays, namely: near-eye augmented reality (AR) and virtual reality (VR) displays, have received much attention in recent years as next-generation displays that present a “live” visual experience to the viewer. ing. In addition, near-eye AR displays are used for moving high-definition 3D content that is mixed with real-world scenes around the viewer to extend the viewer's access on the go. Is considered the ultimate means of presenting to. The main goal of AR displays is to go beyond current mobile display viewing constraints without compromising user mobility and to provide a viewing range that is not limited by the physical constraints of mobile devices. On the other hand, near-eye VR displays are intended to present the viewer with a 360 ° 3D cinema viewing experience that immerses the viewer in the content being viewed. Both AR display technology and VR display technology are regarded as the "next computing platform" following mobile phones and personal computers, which is the growth of mobile users' information access and the information market. And expand the business that provides it. An AR / VR display may be referred to herein as a "near-eye" display to emphasize the fact that the method of the present invention is generally applied to near-eye displays and is not limited to AR / VR displays themselves. Many.

  The main drawbacks of the existing near-eye AR and VR displays are: motion sickness-like sensations caused by low refresh rate display technology; And nausea; and achievement of limited eye resolution in a reasonably wide field of view (FOV). Existing attempts to address these shortcomings include: using a higher refresh rate display; using a higher pixel (higher resolution) display; or using multiple displays or screens. A common theme in all of these attempts is the need for higher input data bandwidth. New compression methods are needed to handle higher data bandwidths without adding bulk, complexity, and excessive power consumption to near-eye displays. Although the use of compression is the usual solution for processing large amounts of data, the requirements for near-eye displays are unique and exceed what can be achieved with traditional video compression algorithms. Video compression for near-eye displays needs to achieve higher compression ratios than provided by existing compression schemes, and adds the requirements of extremely low power consumption and low latency.

  The near-eye display's high compression rate, low latency, and low power consumption constraints are new to data compression, such as compression capture and display, and data compression schemes that take advantage of the capabilities of the human visual system (HVS). Need a different approach. It is therefore an object of the present invention to introduce a near-eye compression method that overcomes the limitations and weaknesses of the prior art, and thus can meet the stringent requirements of mobile device design for compactness and power consumption, and It is possible to produce a near-eye display that can provide the user of the device with an enhanced visual experience of 2D or 3D content over a wide angular range. Further objects and advantages of the present invention will become apparent from the following detailed description of the preferred embodiments of the invention, which description proceeds with reference to the accompanying drawings.

  There are numerous prior art techniques that describe methods for near-eye displays. As a typical example, Non-Patent Document 1 describes an augmented reality (AR) display. The near-eye display prototype described utilizes an LCD to reproduce the light field with multiple layers stacked, but this does not address the requirements of data compression and low latency. The AR display also achieves an unobstructed format with a wide field of view, allowing mutual occlusion and depth of focus cues. However, the process of determining the LCD layer pattern is based on time and power consuming, computationally intensive tensor factorization. The AR display also has a significant reduction in brightness due to the use of shaded LCDs. This remains another question of how display technology affects the performance of near-eye displays and how prior art technology is inadequate in solving all the problems presented in the near-eye display field. It is an example.

  The prior art near-eye display system 100 shown in FIGS. 1a, 1b includes a processor, which can be an embedded processor 102 or an external processor 107, an eye and head tracking element 210, a display device 103, and a display image magnification and human. It consists of a combination of several elements, such as optics 104 for relay to a vision system (HVS) 106. The processor 102 (FIG. 1a) or 107 (FIG. 1b) processes the sensory data obtained from the eye and head tracking element 210 and produces a corresponding image that the display 103 will display. This data processing may be performed by the embedded processor 102 (FIG. 1a) within the near-eye device, or such processing may be performed remotely by the external processor 107 (FIG. 1b). The latter approach uses a more powerful processor, such as the latest generation CPUs, GPUs, and task-specific processing devices, to process the incoming tracking data and render the corresponding images in a personal area network (Personal Area Network). PAN) 108 to the near-eye display 109. The use of an external processor allows the system to use a more powerful image remote processor 107, which has the processing throughput and memory necessary to perform image processing without burdening the near-eye display 109. Have. On the other hand, transmitting data via the PAN has its own problems such as low latency and high resolution video transmission bandwidth. A new low-latency protocol for video transmission protocols in PAN (see Non-Patent Document 2) may enable the use of an external processor 107 for near-eye display image generation, which results in high quality. While enabling immersive stereoscopic VR displays, such PAN protocols meet the high bandwidth data requirements for a new generation of near-eye AR and VR displays aimed at presenting high resolution 3D and VAC-free viewing experiences. I can't handle it.

Maimone, Andrew, and Henry Fuchs. "Computational augmented reality eyeglasses." In Mixed and Augmented Reality (IMSAR), 2013 IEEE International Symposium on, pp. 29-38. IEEE, 2013 Razavi, R .; Fleury, M .; Ghanbari, M .; , "Low-delay video control in a personal area network for augmented reality," in Image Processing, IET, vol. 2, no. 3, pp. 150-162, June 2008

  Nevertheless, the use of more sophisticated display technologies poses new challenges to the overall system. The new image generation method needs to generate more data to send to the display, and due to the size, memory and latency constraints of the near-eye display, the traditional compression used to process large amount of data. The method is no longer appropriate. Therefore, new methods are needed to generate data, compress it, and send it to a near-eye display.

  In the following description, like drawing reference numbers are used for like elements, even in different drawings. Matters defined in this description, such as detailed structures and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, the invention may be practiced without these explicitly defined matters. In addition, known functions or structures will not be described in detail because they obscure the present invention with unnecessary details. In order to understand the present invention and to confirm the way in which the present invention can be practiced, some embodiments of the present invention will be described with reference to the accompanying drawings by way of non-limiting examples only.

FIG. 1a shows a block diagram of a prior art near eye display system incorporating an embedded processor. FIG. 1b shows a block diagram of a prior art near-eye display system incorporating an external processor connected. Figure 2a shows a block diagram of a near-eye display system of the present invention with an embedded processor. FIG. 2b shows a block diagram of the inventive near-eye display system with an external processor. FIG. 3a shows a functional block diagram of an encoder applying the visual decompression function of a compressed display in the context of the near-eye display system of the present invention. FIG. 3b shows the basis coefficient modulation of the visual decompression method of the present invention. FIG. 3c illustrates the truncation of basis coefficients for the visual decompression method of the present invention. FIG. 4a illustrates the field of view (FOV) area around a viewer's point of gaze used by the foveated visual decompression method of the present invention. FIG. 4b shows a block diagram of a near-eye display system incorporating the foveated centralized visual defrosting method of the present invention. FIG. 4c shows the truncation of the basis coefficients of the “foveated focused visual decompression” method of the present invention. FIG. 5a shows an implementation of a light modulator element of a near-eye display system adapted to the viewer's HVS angular vision and FOV. FIG. 5b shows the optical element implementation of the near-eye display system of the present invention. FIG. 6a illustrates a multifocal plane embodiment of a near-eye light field display of the present invention. FIG. 6b illustrates an embodiment of the present invention implementing a multifocal near-eye display with a standard horopter surface. FIG. 7 illustrates the generation of content for the multifocal plane near-eye light field display of the present invention. FIG. 8 shows an embodiment implementing the multifocal plane depth filtering method of the present invention. FIG. 9 illustrates an embodiment implementing a compressed rendering of light field data input to the multifocal near-eye light field display of the present invention.

  In the following detailed description of the invention, reference to “one embodiment” or “an embodiment” refers to certain specific features, structures or structures described in connection with the embodiment. A feature is meant to be included in at least one embodiment of the invention. In many places in this detailed description the phrase "in one embodiment" is found, but not all are necessarily referring to the same embodiment.

  Presenting near-eye display viewers with a high-resolution and wide-field-of-view (FOV) 3D viewing experience requires a display resolution of 8 megapixels per eye, which is close to the visible limit of the eye. The resulting increase in display resolution imposes some requirements on the overall near-eye display system, the most difficult of which is the increase in data interface bandwidth and processing throughput. The present invention introduces a method for addressing any of these difficulties in near-eye display systems by using a Compressed Display system (defined below). 2a and 2b are block diagram illustrations of a near eye display system 200 using the method of the present invention. In FIG. 2a, which illustrates one embodiment of the near-eye assembly 201 of the near-eye display system 200, a newly designed element, encoder 204, has been added to the near-eye display system 200, which is a QPI solid state imager-based display. It serves to compress the data for a compressed display 203 such as (QPI imager display in the drawing) (US Pat. No. 7,767,479 and US Pat. No. 7,829,902). In addition to the QPI imager where each pixel emits light from a different color solid state LED or laser emitter, it emits light from different color solid state LEDs or laser emitters arranged side by side, wherein a plurality of solid state LEDs are provided. Alternatively, imagers are known in which the laser emitter functions as a single pixel. Such devices of the present invention are commonly referred to as emissive display devices. Further, the present invention can be used to form a light source for many types of spatial light modulators (SLMs, microdisplays) such as DLPs and LCOSs, and also as backlight sources for LCDs. Can be used. The terms “solid state imager display”, “display element”, “display” and like terms are often used herein to refer to the compressed display 203. In FIG. 2b, which illustrates another embodiment of the near-eye assembly 205 of the near-eye display system 200, the encoder 204 has the same function as that of FIG. 2a, except that it is part of an external data source driving the near-eye assembly 205 remotely. As you meet. FIG. 2b shows an external data source comprising an external processor 207 and an encoder 204, the latter being connected to a near-eye display assembly 205 via a wireless link 208, such as a wireless personal area network (PAN), or via wiring 209. Shown as connected. In any case, the encoder 204 utilizes the compression processing function of the solid state imager display 203 to achieve a high compression rate while generating a high quality image. The encoder 204 also utilizes the sensory data provided by the eye and head tracking design element 210 to further increase the data compression gain of the near eye display system 200.

Definitions "Compressed (Input) Display" allows to directly display the content image of the provided compressed data, directly input in compressed format, without first decompressing the input data. A display system, subsystem, or element. Such a compressed display can modulate an image at a high sub-frame rate against a higher order basis for direct perception by the human visual system (HVS). With such a display feature, referred to as "visual decompression", as defined below, a compressed display allows a discrete cosine transform (DCT) or discrete Walsh to be integrated and perceived by the HVS as a decompressed image. Higher-order macros containing (n × n) pixels can be directly modulated using an expansion coefficient of a transform (Discrete Walsh Transforms: DWT) (US Pat. No. 8,970,646).

  "Dynamic Gamut" -Compressed display systems may also include a feature known as Dynamic Gamut (US Pat. No. 9,524,682), where the display system is a frame header. Using the word-length adjusted (compressed) gamut data provided within, the gamut can be dynamically adjusted on a frame-by-frame basis. In using the dynamic color gamut feature, the compressed display system processes the input data with a compressed color gamut that matches the input frame image and the visual acuity of the HVS to modulate it into a corresponding image. Both visual decompression and dynamic color gamut capabilities of the compressed display eliminate the need for decompression of input data, reducing interface bandwidth and processing throughput on the display side. Supported by Compressed Display.

  "Visual Decompression" takes advantage of the inherent perceptual power of HVS to display the compressed visual information directly by the display, rather than displaying the decompressed visual information after the first decompression. There are a number of compression visual information modulation methods that allow. Visual decompression reduces the interface bandwidth to the display and the processing throughput required to decompress compressed visual information.

  Visual Decompression-FIG. 3a shows a functional block diagram of an encoder 204 (of FIGS. 3a and 3b) that applies the visual decompression capabilities of the compressed display 203 in the context of the near-eye display system 200 of the present invention. The input image 301 generated by the processor 202 or 207 is first transformed by the visual decompression transformation element 302 into a known higher order basis, eg a DCT or DWT basis. The quantizer 303 then quantizes the selected subset of these higher order basis coefficients obtained. Similar to typical compression schemes using DCT and DWT, such as MPEG and JPEG, the visual decompression applied by the encoder 204 of the near-eye display system 200 of the present invention, in part, produces a subset of the bases with low frequencies. Achieving compression gain by truncating the high frequency basis while selecting. In one embodiment of the invention, quantizer 303 uses a quantization step size that is the same as the quantization step size for the quantization of the selected subset of basis coefficients. In another embodiment of the invention, the quantizer 303 takes advantage of the capabilities of the human visual system (HVS) and uses a larger quantization step for high frequency coefficients to make the coefficients less perceptible to HVS. Reducing the associated data transfer bandwidth, which in practice adapts the capabilities of HVS to achieve higher visual decompression gain. The quantized coefficients are then time (or time division) multiplexed by a run length encoder 304, which visually decompresses the set of coefficients associated with one of the selected bases. Send to the corresponding compression display 203 at once, and the visual decompression-enabled compression display 203 then modulates the received coefficients as the size of the associated basis displayed by the visual decompression-enabled compression display 203. Compressed display 203 modulates one of the bases at a time within one video subframe, and thus the modulated base is timed beyond the time constant of the HVS impulse response (typically ~ 5ms). Are not separated. For example, if eight bases are selected to transform the input image 302, a 60 Hz (16.67 ms) video frame is segmented into subframes of ~ 2 ms, which is significantly smaller than the time constant of the HVS impulse response. One basis coefficient is modulated by the compressed display 203 during a subframe.

  In another embodiment, visual decompression transform block 302 extracts DWT and DCT coefficients directly from externally provided compressed input data formats, such as MPEG and JPEG data formats, and quantizes the extracted DWT and DCT coefficients. Provided to the container 303. In this case, the quantizer 303 uses a larger quantization step to further enhance the DWT and DCT coefficients of the MPEG and JPEG data formats to reduce the data transfer bandwidth associated with the coefficients that HVS is less likely to perceive. And thereby achieve higher visual decompression gain by adapting HVS capabilities.

  In another embodiment of the invention, the transform 302 and quantized 303 basis coefficients of the input image 301 are field sequenced 304 directly into a compressed display 203 that can modulate the data directly compressed for HVS. (See compressed display definition above). The visual decompression gain achieved by the display 203, in addition to reducing the memory requirements on the display 203, provides for direct conversion and modulation of compressed image data from the processor 202 or 207 to the display 203 and then to the HVS 106. The latency when transferring data is also reduced. Such reduction of latency in near-eye display systems is extremely important to reduce viewer discomfort, which is typically detected by the eye and head tracking sensor 210 of the viewer. It is caused by an excessive delay of the input image 301 with respect to the line-of-sight direction. The reduced latency is due to the method of direct conversion and modulation of compressed image data, wherein a subset of basis coefficients are received in a subframe time sequence, which is typically shorter than the HVS time constant. The HVS 106 is continuously modulated in time with respect to the HVS 106 by 203, so that the HVS 106 begins to partially integrate the subset, and the image input 301 to some of the subframes of the modulated basis coefficient. This is because the feedback delay in incorporating the gaze direction information sensed by the eye and head tracking 210 into the input image 301 is significantly reduced. Also, in this method of directly converting and modulating compressed image data, the latency is reduced because the compressed input image 301 generated by the encoder 204 and represented by the selected basis coefficient is the data of the input image 301. Because it is displayed directly to the HVS 106 without the processing delay typically introduced by prior art systems of first compressing on the processor 102 or 107 side and then decompressing on the display 203 side. In addition to reducing the latency of the near-eye display system, the above-described near-eye visual decompression method of the present invention that directly transforms and modulates compressed image data also significantly reduces the processing, memory and power consumption requirements of the near-eye system. . This is because the above method eliminates the processing related to the compression of the input image 301 data on the processor 102 or 107 side and the decompression on the display 203 side. It is worth mentioning that the above-described near-eye visual decompression method of the present invention that directly transforms and modulates compressed image data takes advantage of the inherent capabilities of HVS 106 for perception by temporal integration of visual sensation. Achieve reduced latency and processing requirements. That is, the above-described near-eye visual decompression method of the present invention that directly transforms and modulates compressed image data achieves reduced latency and processing requirements by adapting the capabilities of HVS.

FIG. 3b shows the base coefficient modulation of the visual decompression method for the near-eye display system of the present invention. Instead of the row / column selection method typically used in current displays to address (and modulate) individual display pixels, in the near-eye visual decompression method shown in Figure 3b, the display is of higher order. A group of (n × n) pixels representing the basis w _ij is modulated with the same basis coefficient value C _ij . Within a subframe of the video input image 301, the near-eye compressed display 203 addresses a block of (n × n) pixels with the associated basis coefficient C _ij as a macro representing the basis element W _{ij of the} display. . The temporal sequence of basis coefficient modulation sub-frames within a video frame is integrated by HVS in chronological order, which leads to a gradual perception of the input image 301 within the duration of the video frame. As can be seen from FIG. 3b, the near-eye compressed display 203 must have a response time and modulation capability to receive and modulate the base coefficients at the subframe rate, which subframe rate is the number of video frame rates. Double speed, for the example with 8 subframes described above, the visual decompression subframe rate would be 8 × 60 Hz = 480 Hz. In one embodiment of the present invention, the near eye compressed display 203 is implemented using a solid state imager due to the high speed image modulation capabilities of the solid state imager. In addition to the solid state imager's ability to support the visual decompression method of the present invention, the near eye display system 200 of the present invention benefits from its small size (compactness), low power consumption, and brightness provided by the QPI 203. Also, the volume rationalized near-eye display system 200 is realized.

Referring again to FIG. 3a, the quantizer 303 truncates the basis constants calculated by the visual decompression transform element 302 based on a given truncation criterion, and then a selected subset of basis coefficients to a given subset. Quantize to a given word length based on a quantization criterion. FIG. 3c shows the basis coefficient truncation performed by quantizer 303 on a (4 × 4) visual decompression basis. As shown in FIG. 3c, the quantizer 303 truncates the set of 16 basis coefficients by selecting the subset of 8 basis coefficients marked in FIG. 3c. The criterion for this selection is to discard high frequency basis coefficients that exceed the HVS temporal vision limit, and the relatively high index basis is shaded in FIG. 3c. For the selected subset of basis coefficients, quantizer 303 then truncates the corresponding word lengths received from visual decompression transform 302 to fewer bits, eg, an 8-bit word. It should be noted that the visual decompression transform 302 typically has a longer word length, eg, 16-bit words, to perform the transform computation. In another embodiment, quantizer 303 truncates the selected subset of basis coefficients with different word lengths for different basis coefficients. For example, referring to FIG. 3c, the low frequency coefficient C ₀₀ is quantized to 8 bits, and the remaining base coefficients along the row coefficient C _0j and the column coefficient C _i0 are successively smaller in word length, eg 6 bits each. It is quantized using 4 bits and 2 bits. Both the base coefficient truncation and its word length quantization criteria are fixed and either known a priori by the display 203 or signaled to the embedded display 203 via the data stream ( Or it is communicated). The compression gain of the data transfer bandwidth expected to be achieved by the near-eye visual decompression method of the present embodiment is typically determined by the number of dimensions of the base used for transfer of the input image 301 and the quantizer 303. Depending on the base coefficient truncation criteria used, it is typically 4 to 6 times, which means that the image data transfer bandwidth from the processor 102 or 107 to the display element 203 is of the present embodiment. By the visual decompression method described above, it is meant to be reduced by a factor in the range 4 to 6 times. It should be noted that the visual compression gain of this embodiment is achieved by adapting the display to the HVS temporal vision.

  Dynamic Gamut-In another embodiment of the invention, the near-eye display system 200 utilizes two factors that provide the opportunity for additional visual decompression: (1) The gamut of a video frame is typical. The color gamut of a preset standard display (eg NTSC) (where the color coordinates of the pixels of the display within the standard gamut above are typically represented in 24-bit words (8 bits per primary color)). (2) The color vision in the peripheral area of HVS is significantly lower than that in the central visual area. In this embodiment, the visual decompression transform block 302 includes in the header of each input video frame the color coordinates of the frame gamut primaries along with the color coordinates of each pixel in the frame expressed relative to the frame gamut primaries. The received frame header is sent to the quantizer 303. The visual decompression transform block 302 then sends the frame gamut header it receives to the quantizer block 303, along with the set of higher order basis coefficients it extracts. The quantizer block 303 then proportionally truncates the word length in the image frame gamut representing the color coordinates of each pixel in the image frame to less than the default of 24 bits (8 bits per color). Utilizing the reduced size, a smaller transmitted gamut size of the frame, relative to the standard gamut size of the display, uses a word length smaller than the default of 24 bits for each received image frame. The color coordinates of pixels can be expressed. The visual decompression block 302 also conveys, within each input video frame header, the color gamuts and coordinates of the plurality of image areas within the image frame within each frame image area within the frame header for that frame image area. It can also be received with the color coordinates of each pixel expressed in terms of the gamut primaries. In this case, the quantizer block 303 proportionally truncates the word length representing the color coordinates of each pixel in each frame image area to less than the default of 24 bits (8 bits per color). For a typical video frame image, the size of the image frame data that needs to be sent to the compressed display 203 can be reduced by a factor of 2 to 3 by either of the two methods described above, the latter method being the upper limit of this range. Achieves a compression factor close to. Upon receipt by a compressed display 203 having the ability to dynamically adjust the gamut of a frame or frame image area, as defined above, the compressed display 203 receives the frame or frame transmitted in the received header. Pixels in a received (truncated) frame or frame subregion after combining the gamut of the transmitted frame or frame subregion using its original primaries using coordinate data of the color gamut of the frame region By modulating the color coordinate data of the above, light representing each pixel of the frame or frame sub-region is generated. It should be noted that the visual compression gain of this embodiment is achieved by matching the display gamut to the image frame gamut.

  Foveated Focused Visual Defrosting-FIGS. 4a, 4b illustrate yet another visual defrosting method for the near-eye display system 200. FIG. In the present embodiment shown in FIGS. 4 (a) and 4 (b), the line-of-sight direction (axis) 401 and the focus direction of the viewer, which are based on the inter-pupilary distance (IPD) of the viewer, are the eye and head. By sensing and tracking by the tracking element 210 and using it to apply different visual decompression basis coefficient truncation and quantization criteria to different regions of the image displayed within the viewer's field of view (FOV) 420. , Utilizing the HVS angle (visual acuity) distribution of visual perception while effectively achieving the best possible visual perception within the FOV region where the viewer's eyes are in focus (402) Regularly achieves a high level of visual compression over the remaining regions 403-412 of the viewer's FOV, which is gradually decreasing. In effect, in this embodiment visual decompression is applied to match the angular distribution of HVS visual acuity with a compressed word length that is proportional to the angular distribution of the viewer's visual perception across the FOV.

  FIG. 4a illustrates the method of this visual decompression embodiment, which is referred to herein as "foveolar focused visual decompression," in which the viewer's spatial vision (angular vision) is Highest in the in-focus (402) region (foveal region of the retina) and falls regularly across the rest of the viewer's FOV 403-412 (parafovea 403-406 and parafoveal regions 407-412 of the retina). To achieve the highest visual perceptual ability in the region where the viewer is in focus (402) while achieving higher visual decompression gain. In this embodiment, the cues of the viewer's eye focus and gaze direction 401 are the sensory data provided by the sensors of the eye and head tracking elements 210 by the foveated centralized quantizer 430 of FIG. 4b. Gaze direction for each eye is determined by the position of the pupil of the eye in the reference head detection frame as detected by the sensor of the eye and head tracking element 210, for example. Similarly, the viewer's focal length (or the eye separation distance, defined as the distance at which the viewer's eyes are in focus and focus) of the near-eye display system is detected by a sensor in the eye and head tracking element 210. Determined by the relative interpupillary distance (IPD) between the centers of the two pupils of the viewer. For the region of the FOV 420 that the viewer focuses (402), which typically covers the foveal region of the retina when the eye lens of the viewer is in focus, the highest image resolution is , A foveal-concentrated quantizer 430 to select a subset of basis coefficients as large as possible and quantize this selected subset of basis coefficients with the largest possible word length. It For the remaining regions of the viewer's FOV 420, regions 403- 412 of FIG. 4a, the foveated centralized quantizer 430 selects a smaller subset of basis coefficients and uses this with a smaller number of bits. Quantize the base coefficient. In applying such a basis coefficient truncation and quantization criterion, the foveated focused visual decompression method of the present embodiment achieves the highest resolution within the region 402 of the focus of the viewer, and the rest of the FOV 420 of the viewer. Across the regions 403-412, one achieves a fairly high visual decompression gain over these FOV regions while achieving a lower regularity without degrading the perception of the viewer. It should be noted that the term "foveated" as used within the context of this embodiment transfers the display resolution from the center of the fovea of the viewer's eye to the peripheral region of the retina of the outer viewer's eye. It is intended to refer to the onward, HVS visual acuity profile (distribution). Such image resolution depending on the direction of the viewer's line of sight is known in the prior art as "foveated rendering", an example of which is Guenter, B. et al. Finch, M .; Drucker, S .; , Tan, D.M. , And Snyder, J .; , "Foveated 3D Graphics", ACM SIGGRAPH ASIA, Nov. , 2012, which typically concentrates the image input 301 in the fovea by image rendering, potentially reducing the computational load of image rendering on the processor 102 or 107. These benefits are not directly translated into the bandwidth of the image interface 301 and the reduction of decompression computational load on the display 203 that can be achieved by the above-described foveated centralized visual decompression method of this embodiment.

  FIG. 4b shows a block diagram of a near-eye display system using the foveated centralized visual defrosting method of the present invention. Referring to FIG. 4b, the foveated centralized quantizer 430 following the visual decompression transform 302, when the viewer's focal length is known based on the input provided by the eye and head tracking element 210, The base truncation and quantization are chosen to be adapted as follows: the displayed image area (focused by the eye onto the foveal region of the viewer's retina), corresponding to the viewer's focal region 402. Image region) has the highest spatial resolution, and the remaining regions 403-412 of the viewer's FOV 420 are the angular visual acuity (spatial visual acuity) of the viewer's eye over the para-foveal and peri-fovea of the viewer's retina. Has a lower resolution that is consistent (or proportional) with the gradation of, and regularly. FIG. 4c shows an example of the base truncation and quantization selection of the foveated centralized quantizer 430 according to the foveated centralized visual decompression method of the present invention. FIG. 4c shows an example of truncation of the basis coefficients performed by the fovealized quantizer 430 for a (4 × 4) fovealized visual decompression basis. As shown in the example of FIG. 4c, the foveated centralized quantizer 430 has a maximum of eight basis coefficients marked in the first panel of FIG. 4c, corresponding to the focal region 402 of the viewer. Truncate the set of 16 basis coefficients by selecting a subset of For such a region (402), the foveal focused quantizer 430 also uses the maximum quantized word length, eg, 8 bits per color, to select the basis for the region 402 of the viewer's FOV. Express the coefficient. As shown in FIG. 4c, for the peripheral focal region 403, the fovea focused quantizer 430 sets the set of 16 basis coefficients to a smaller number of 7 basis coefficients, appropriately marked in FIG. 4c. Truncate to a subset. For such regions, the fovea focused quantizer 430 also selects a relatively short word length, eg 7 bits or 6 bits, to represent the selected basis coefficients for the region 403 of the viewer's FOV. You can do it. As shown in FIG. 4c, for the outer peripheral regions 404-412, the foveated quantizer 430 sets the set of 16 basis coefficients to a regularly smaller number of basis coefficients, appropriately marked in FIG. 4c. Truncation into subsets and shorter word lengths, eg less than 6 bits, may be selected to represent the selected basis coefficients for the region 403 of the viewer's FOV.

Referring again to FIG. 4 b, the truncated and quantized basis coefficients generated by the foveated quantized quantizer 403 for multiple FOV (200) regions are further encoded by a run length encoder 435. The run-length encoder 435 embeds a control data packet (or data header) in the encoded data stream, the control data packet including which basis coefficient is included in the streamed data, and its truncation and quantization word. The length is signaled (or specified). For example, within the data field designated for transmitting the base coefficient value C _ij , the run length encoder 435 may specify a data field that specifies whether the base coefficient value C _ij is included and the associated quantized word length. Add a header that contains. The added basis coefficients are then transmitted at one time to the compression display 203 as a set of time division multiplexed coefficients for one of the selected bases, which is followed by a run display. The control header added by the length encoder 435 is decoded and the received coefficient is modulated accordingly as the associated basis magnitude displayed by the compressed display 203. As shown in FIG. 4c, the number of basis coefficients associated with the display areas 403- 412 is regularly reduced, so the resolution of the displayed image is also typical of HVS visual acuity over these areas of the displayed image. It is regularly reduced in proportion to the distribution of. As mentioned above, the criteria for selecting the basis coefficients to be included for each of the display areas 403-412 are based on and correspond to the angular visual acuity (spatial visual acuity) of the corresponding retina areas. Is set as a design parameter of the fovea centralized quantizer 430.

  The data transfer bandwidth compression gain, which is expected to be achieved by the near-eye fovea centralized visual decompression method of the present invention, is typically the number of basis dimensions used to transform the input image 301, as well as the fovea. Depending on the basis coefficient truncation and quantization criteria used by the lumped quantizer 430, it is typically higher than the data transfer bandwidth compression gain of the visual decompression method described above. Knowing that when the eye is in focus, the displayed image area 402 nominally extends over the angular range of the foveal region of the viewer's eye (about 2 °), the near-eye display system 200 When the total FOV is, for example, 20 °, the foveated focused visual decompression method of the present invention achieves a compression gain of 4-6 times in the displayed image area 402, and also displayed image areas 403- 412. Regularly achieve higher compression gains over. Using the example base coefficient truncation shown in FIG. 4c, the compression gain achieved is 8/7 times for regions 403, 404 and 8/5, 8/3 for regions 405, 406, respectively. , And eight times for the peripheral areas 407-412. Considering the area of each image region 401 to 412 with respect to the FOV of the displayed image, and the overhead due to the additional data added by the run length encoder 435, the center of the invention with respect to the foveated centralized basis coefficient truncation example of FIG. 4c. The combined compression gains achievable with the foveated visual decompression method are 24 to 36 times, which means that the image data transfer bandwidth from the processor 102 or 107 to the display element 203 is the foveated focused image of the present invention. It means that the visual decompression method reduces the amount by 24 to 36 times. It should be noted that if the FOV of the near-eye display system 400 of the above example is greater than 20 °, for example 40 °, the compression gain achieved for the peripheral regions 407-412 is achieved in the central regions 402-406 of the image. The compression gain asymptotically becomes eight times higher than the compression gain. For large display FOVs, the peripheral image area constitutes the majority of the displayed FOV, so the foveated centralized visual decompression method of the present invention uses a near-eye display system 200 with an HVS FOV (HVS FOV of 100 °). Higher (> 40 times higher) composite compression gain can be achieved with an FOV close to (known to exceed 1).

  In another embodiment of the fovea focused visual decompression method of the present invention, the visual decompression transform 302 includes image regions corresponding to the fovea 402, parafovea 403-406 and peri-fovea 407-412 regions of the retina of the eye. With respect to, higher compression gain is achieved by using different values of higher order basis. In this embodiment, the visual decompression transform 302 receives an input of an eye gaze point (orientation) 401 from the eye and head tracking element 210, followed by a foveal region 402, parafoveal regions 403-406 and a circumference. After identifying the image regions corresponding to the foveal regions 407-412, different higher order basis values are used to generate a transformed version of each image region. For example, the visual decompression transform 302 uses the (4 × 4) basis for generating transformed versions of image regions 402-406 and (8) for generating transformed versions of image marginal regions 407-412. Use the basis of x8). The visual decompression transform 302 then stitches the transformed images of the multiple regions together to form a composite transformed image with embedded control data identifying the order of the basis used for each image region. Transmit to the central foveal quantizer 430. The foveated centralized quantizer 430 applies an appropriate truncation of basis coefficients and a quantization criterion to each image region and then sends the image and corresponding control data to a run length encoder for transmission to the compressed display 203. Send to 304. By using a higher order basis in the image area corresponding to the foveal marginal area, the foveated focused visual decompression method of this embodiment will be able to achieve even higher compression gains. Using the (4 × 4) basis for the image regions 402-406 and the (8 × 8) basis for the image peripheral regions 407-412 for the above example, the fovea of this embodiment The centralized visual decompression method can achieve a compression gain that is asymptotic to 16 times that achieved in the central image regions 402-406. Thus, the foveated focused visual defrosting method of this embodiment is a composite of 32 to 48 times for the above example of a 20 ° display FOV and possibly 64 times for a 40 ° display FOV. It is possible to achieve various compression gains.

  The above-described levels of compression gain that the foveated centralized visual decompression method of the present invention can achieve directly translates to reduced processing and memory on the side of the display 203, which directly translates into power, volume, and cost savings. . It should be noted that the processing and memory requirements of the visual decompression block 302 and the foveated centralized quantizer 430 block of FIG. 4c are equivalent to the processing and memory requirements of conventional image decompression elements, while the latter increases the image data bandwidth. However, this causes a large increase in the processing and memory requirements of the display 203 and a proportional increase in power consumption. Furthermore, the processing and memory requirements of the visual decompression block 302 and the centralized quantizer 430 block of FIG. 4c are comparable to the processing and memory requirements of the conventional centralized rendering block and thus the fovea of the present invention. The near-eye display system 200 using the centralized visual decompression method is significantly less processing than the prior art near-eye display system of FIGS. 1a, 1b, which incorporates prior art centralized centralized rendering and uses conventional compression techniques. And memory (thus reduced cost and power consumption). The foveated focused visual decompression method of the present invention adapts such gains to the inherent capabilities of HVS, ie, temporal integration of HVS and graded (ie foveated) spatial (angular) resolution (visual acuity). Also note that it is obtained by Also, the level of compression gain that the foveated focused visual decompression method of the present invention can achieve is highest when the near-eye display system 200 is required to display a multi-view or multi-focal light field. It is also important to keep in mind. This is because the processing, memory and interface bandwidth of the system is directly proportional to the number of fields of view or the number of focal planes (surfaces) that the system is required to display. For various designs of near-eye display systems, there may be 6 to 12 views that need to be displayed to achieve a 3D perception level acceptable by the viewer of the near-eye display.

  Foveated Focused Dynamic Gamut-In another aspect of the dynamic gamut embodiment described above, the visual decompression block 302 receives information from the eye and head tracking element 210 regarding the viewer's gaze direction, Visual decompression block 302 then maps the information to corresponding pixel (macro) spatial coordinates in the image frame that identify the center of the viewer's field of view, which visual decompression block 302 maps to the quantizer block. Add to the image frame data sent to 303. With the identified spatial coordinates in the center of the viewer's field of view, the quantizer block 303 then applies a typical HVS (angular or directional) color vision profile to image pixel (or The default 24 bit (8 bits per color) word length of the macro) color coordinates is proportional to the position of each pixel (or macro) relative to the spatial coordinates of the center of the viewer's field of view identified for that frame. Round off. A typical HVS (angular or directional) color vision profile (distribution) is maintained by the quantizer block 303 as a look-up table (LUT) or generation function, which is the viewer. A quantization factor for the word length of the pixel (or macro) color coordinates is identified according to the spatial distance of the pixel (or macro) from the center of the field of view. The LUT or generation function of such a HVS color vision profile is based on a typical viewer's (angular or directional) HVS color vision profile, and depending on the preference of each particular viewer, It can be adjusted or biased by a given factor. The run-length encoder 304 then adds the gamut distribution corresponding to the HVS color vision profile to the quantized color values of the pixel (macro) before sending it to the display element 203 for modulation. The above method of truncating the word length of the color coordinates of a pixel (or macro) based on the angular or directional color vision profile around the center of the viewer's field of view identified for each frame is effectively displayed. Foci centralization of the colors of the image, which leads to a reduction of 2-3 times the size of the image frame data to be sent to the display 203. Because display 203 is a compressed display, the truncated color coordinates of the pixels (or macros) received by display 203 are used directly to modulate the image frame. As used within the context of this embodiment, the term "fovea focused" refers to a display color gamut of the HVS that moves from the center of the fovea of the viewer's eye to the peripheral region of the retina of the outer viewer's eye. It is intended to refer to fitting to the vision profile (distribution). It should be noted that the visual compression gain of this embodiment is achieved by adapting the display to the distribution of HVS color perception vision.

  Near Eye Illumination Field Display-When sending different perspectives of the image or video information of a scene to each eye, the viewer's HVS fuses both images and differentiates between the right and left images or video frames ( The depth conveyed by parallax can be perceived (3D perception), a capability known as stereoscopic depth perception. However, in a conventional 3D display, which typically uses two viewpoints (one for each eye), the depth perceived by the viewer is different from the depth at which the viewer's eye is in focus. There are cases. This creates a discrepancy between the focus provided to the viewer's HVS and the adaptive depth cue (an effect known as adaptive eye separation disagreement (VAC)), which can lead to viewer headaches, discomfort and eye strain. Fatigue may be caused. The VAC provides each eye of the viewer with a proportional perspective of the total light field so that the viewer's HVS can naturally fit and focus at the same point in the light field, or focus. It can be eliminated by a light field that can be adjusted. The perspective of the light field presented to each of the viewer's eyes can be the angle or depth sample (or slice) of the light field. If the perspective presented to each of the viewer's eyes is an angular sample of the light field, this approach is called a multi-view light field, and if depth samples are used, the approach is a multifocal plane light field. Called the field. Although the details of these implementations may differ, the above two approaches to presenting a VAC-free light field to the viewer's HVS are functionally equivalent representations of the light field. With either approach, the bandwidth of the visual data presented to the viewer's HVS is proportional to the number of light field samples (viewpoints or focal planes) used to represent the perspective of the light field. Therefore, it is much larger than the conventional stereoscopic method that presents one viewpoint (or perspective) to one eye. The increase in bandwidth of visual data leads to a proportional increase in processing, memory, power, and volume aspects of near-eye display systems, which eliminates VAC using the light field principle described above. It will be even more difficult to realize a near-eye display. The following paragraphs show the compactness required for a practical near-eye AR or VR display system using the above-mentioned light field principles by applying the above-described visual decompression method and other HVS vision adaptation methods. The VAC can be eliminated to provide the viewer with a high quality visual experience while achieving a (streamlined look).

  Near Eye Light Field Modulator-In one embodiment of the invention, a group of multiple physical pixels in each of the display (or light modulator) right element 203R and left element 203L of the near eye display 200 is used to create a light field of the light field. The sample (viewpoint or focal plane) is presented to the viewer's HVS (or modulated by a near-eye display system). Here, a plurality of (m × m) physical pixel groups of such light modulator elements 203R, 203L are collectively referred to as “(m × m) modulation group” or “macro pixel”. For brevity, the individual (individual) physical pixels of the light modulator elements 203R, 203L are called macropixels (or m-pixels) and are used to modulate the light field sample (viewpoint or plane). Macropixels are called M-pixels. In the case of a multi-view light field near eye display system implementation, individual m-pixels, including each M-pixel, are used to modulate (or display) multiple viewpoints of the light field presented to the viewer's HVS. ), And in the case of a multifocal surface (plane) light field implementation, M-pixels are used to represent multiple depth virtual representing the depth plane (sample) of the light field presented to the viewer's HVS. Modulate (or display) the image surface. The dimensionality of M-pixels is represented by (m × m) m-pixels, which represents the total number of light field samples that the near-eye display system will present to each eye of the viewer. In this embodiment, the optical (emission) characteristics of the light modulator elements 203R, 203L of the near-eye light field display 200 are adapted to the viewer's HVS angular acuity and FOV. The HVS angular visual acuity reaches the highest level in the foveal region 402 of the viewer's eye and decreases regularly toward the peripheral regions 403 to 412 of the retina of the viewer's eye. In the foveal region 402 of the eye of the eye and drops regularly towards the peripheral regions 403- 412 of the retina of the viewer's eye. Therefore, by adapting the viewer's HVS angular acuity, the light modulator elements 203R, 203L of the near-eye light field display 200 of this embodiment can be adjusted to the viewer's HVS as described in the following paragraphs. Adapted for angular deep vision.

  FIG. 5a shows an implementation of light modulator (display) elements 203R, 203L of a near-eye display system used to adapt to the viewer's HVS angular vision and FOV. In FIG. 5a, the m-pixel 550 of the light modulator element 203R, 203L is an emissive multicolor photonic microscale pixel (typically 5-10 micrometers in size) with a micro-optical element 555, The micro-optical element 555 directs (or modulates) the collimated light flux emitted from the m-pixels in a given direction within the emission FOV of the light modulator elements 203R, 203L. Also associated with each M-pixel of the light modulator element 203 shown in FIG. 5a is a macro-optical element 560, which directs light emitted from the associated m-pixel to the FOV of the M-pixel. To achieve a given angular density of directionally modulated light flux emitted from the associated m-pixel modulation group. The collimated, directionally-modulated light flux emitted from each m-pixel is referred to herein as a "light field anglet". As shown in FIG. 5a, the dimensionality of the M-pixel is at the highest level at the optical center of the optical aperture of the light modulator elements 203R, 203L, and the depth perception of the HVS away from the image modulation area corresponding to the center of the fovea. It decreases gradually in proportion to eyesight. Also, as shown in FIG. 5a, the angular range (or FOV) of the M-pixel is the narrowest value at the optical center of the optical aperture of the light modulator (display) elements 203R, 203L and corresponds to the center of the fovea. It gradually increases in inverse proportion to the decrease in the angular visual acuity of the HVS away from the image modulation area. As a result, the angular density of the light field anglets is highest in the central region of the light aperture of the light modulator (display) element 203R, 203L and decreases regularly in its peripheral region. In practice, the optical F / # of each of the light modulator elements 203R, 203L shown in FIG. 5a has the highest value in the central region of its light aperture, and from the image modulation region corresponding to the center of the fovea. It gradually decreases in proportion to the distribution of the visual acuity of the distant HVS. Thus, in practice, in this embodiment, the light emitted from the light modulator elements 203R, 203L has its highest resolution, the highest level of visual acuity of the viewer's HVS in the foveal region 402 of the viewer's eye. In order to be available in the target image area, it fits into the HVS visual acuity distribution and regularly decreases towards the peripheral areas 403- 412 of the retina of the viewer's eye. In order to adapt the range of the pupil movement of the viewer's eyes to the viewer's near-field to far-field (~ 7 °), as shown in FIG. 5a, the maximum resolution of the optical modulator elements 203R and 203L is set. The central region (FOV region of the central ± 5 ° in FIG. 5a) is defined as all possible fossa FOV regions 402 of the fovea of the eye within the range of the viewer's eye movement from the near field to the far field of the viewer. Make it wide enough to encompass the location. Having described above a method for realizing a light field modulator that is optically adapted to the vision of an HVS, the following paragraphs describe the optical modulator elements 203R, 203L optically adapted to an HVS as described above. A method for realizing a near-eye light field display 200 using the above-described multi-view or multi-focal surface light field sampling method in combination with the fovea focused visual defrosting method will be described.

  Multi-view light field-FIG. 5b illustrates, at a high level, the coupling between the optical element 206 and the light modulator (display) elements 203R, 203L of the embodiments described above. In the design of optical element 206 of near-eye display system 200 of FIG. 5b, the image modulated by display elements 203R, 203L is appropriately magnified by optical element 206 and relayed to viewer's eye 580. . The optical element 206 can be implemented using a reflector and beam splitter optical assembly, a freeform optical wedge, or an optical waveguide. Although the design details of the design options for these optical elements 206 are different, these design criteria are such that the light modulator (display) elements 203R, 203L are sufficiently enlarged to be relayed to the viewer's eye 580. is there. The design criteria for the (m × m) dimensionality of the selected M-pixels and the effective optical magnification from the micro-optics 555 and macro-optics 560 of the light modulator elements 203R, 203L by the optical element 206 are: The spot size of the M-pixel located in the central optical region of the projector elements 203R, 203L is the minimum viewing distance (near field) of the near-eye display system 200 that covers the central region of the fovea (402-404 in FIG. 4a). As adapted to the HVS (average) spatial vision for the virtual image formed. For example, if the minimum viewing distance of the near-eye display system is 30 cm and the HVS spatial vision at that distance is approximately 40 micrometers, then the M-pixel in the optical central region of the light modulator elements 203R, 203L. Is also 40 micrometers, and if the distance between the m-pixels of the light modulator elements 203R, 203L is 10 micrometers, then the dimensionality of the M-pixels is (4 × 4) m-pixels. Thus, the near-eye light field display system 200 can modulate a maximum of 4 × 4 = 16 viewpoints with respect to the central region (402 to 404 in FIG. 4a) of the fovea of each eye of the viewer. In this example, the viewer reduces the dimensionality of the M-pixels to (3x3), (2x2), and (1x1) m-pixels, as shown in Figure 5a. In the peripheral areas 405 to 412 of the FOV, the reduced number of viewpoints is regularly presented. Thus, the light modulator elements 203R, 203L of this embodiment modulate a relatively large number of viewpoints with respect to the central region of the viewer's fovea (402-404 in FIG. 4a), and the peripheral region of the viewer's FOV. By regularly modulating a reduced number of viewpoints for 405-412, the viewer's HVS aspects of angular vision and depth perception are adapted. This is actually a form of visual compression. The maximum number of viewpoints required to provide the viewer with the highest depth cue is that the viewing is as shown by the viewer's gaze direction and depth of focus as sensed by the eye and head tracking element 210. In the central region of the human fovea (402-404 in FIG. 4a), modulated by the light modulator elements 203R, 203L, in proportion to the typical angular distribution of the HVS visual acuity, the perimeter of the viewer's FOV. This is because, for the regions 405 to 412, the reduced number of viewpoints are regularly modulated. Thus, in the above example, 16 viewpoints are modulated by the display elements 203R, 203L for the central region of the fovea of the viewer's eye (402-404 in FIG. 4a), which is approximately 2 ° wide. Of the FOV peripheral regions 405-412 are modulated so that the bandwidth of the image input 301 is proportional to the total angular width of the FOV of the near-eye display 200 in the center of the fovea of the viewer's eye. It is reduced so that it is primarily proportional to the angular width of the region (402-404 in FIG. 4a). That is, when the width of the FOV of the near-eye light field display 200 is, for example, 20 °, 16 viewpoints are modulated with respect to the central 2 ° width angular region, and on average 4 viewpoints are modulated in the peripheral region, In such a case, an effective bandwidth of about 5 viewpoints is sufficient, which corresponds to a compression gain of 3 times. Of course, if the FOV of the near-eye display 200 is wider than the 20 ° that is assumed in this illustrative example, the visual compression method of this embodiment achieves even higher compression gain.

  Multi-view Light Field Depth Foveated Concentrated Visual Decompression-Regular reduction of the angular (perceptual) visual acuity of the HVS from the central region to the peripheral region of the FOV allows the depth perceptual visual acuity of the HVS to also be viewed. It is regularly reduced from the person's near field (~ 30 cm) to the far field (~ 300 cm). Therefore, HVS requires multiple viewpoints for near-field depth perception than for far-field depth perception. Furthermore, when the viewer's eye is focused and fitted to a particular point, the depth-perceived visual acuity of the HVS is at its highest level in the vicinity of that point and drops regularly as it deviates in depth or angle from that point. To do. Therefore, the viewpoints that contribute to the visual information in the vicinity of the point where the viewer's eyes are in focus and fit, contribute most to the achievement of depth perception, and the number of such viewpoints is the focus of the viewer's eyes. Decreases regularly as the viewer changes from near-field to far-field. Such an attribute of HVS depth perception is yet another visual exploitable by a combination of the (foveated) multi-view light modulator elements 203R, 203L of FIG. Present an opportunity for compression. In an embodiment incorporating both the multi-view light modulator elements 203R, 203L of FIG. 5a and the foveated centralized visual decompression method described above within the near-eye light field display system 200, provided by the eye and head tracking element 210. The sensed viewer focus is used to determine (or identify) a perspective of the light field that contributes most of the visual information in the vicinity of the point where the viewer's eye is in focus. By applying the foveated centralized visual decompression method described above, the viewer's eye focuses the viewpoint of the light field being modulated on the viewer by the multi-view light modulator elements 203R, 203L of FIG. 5a. Proportionally compresses the contributions to the visual information in the neighborhoods that match. Thus, in practice, the method of this embodiment allows the viewpoint of the light field, which contributes most of the visual information in the vicinity of the point where the viewer's eye is in focus, to the multi-view light modulator element 203R of FIG. 5a. , 203L, achieves the best visual perception with the maximum number of modulation basis coefficients with minimally truncated word length representations, while the viewer's eye is in focus. The light field viewpoints with smaller contributions in the vicinity of the above points are used with a smaller number of light field modulation viewpoints separated by wider angular intervals, and in proportion to this, a smaller number of modulations with word lengths truncated. Modulation is performed by the multi-view light modulator elements 203R, 203L of FIG. 5a using the basis coefficients. The net effect of the method of this embodiment is a three-dimensional foveated centralized visual defrosting action in which the visual information near the point where the viewer's eye is in focus is the focus of the viewer. Is modulated with the highest fidelity to match the perceived visual acuity of HVS at, while the visual information in the surrounding areas (front, back and side areas) is away from the point where the viewer's eyes are in focus. Is modulated at a fidelity level that is commensurate with the proportionally reduced perceptual visual acuity of the HVS at different points. The combined method of this embodiment is collectively referred to as multi-view light field depth foveated centralized visual decompression. It should be noted that the term "fovea focused" as used in the context of this embodiment is such that the display resolution is outward from the center of the fovea of the viewer's eye towards the peripheral region of the retina of the viewer's eye. , HVS is intended to be adapted to the depth perception visual acuity profile (distribution).

  Further, in the above embodiments, a greater number of viewpoints are displayed on the display elements 203R, 203L with respect to the central region of the fovea of the viewer's eye (402-404 in FIG. 4a) as shown by the eye and head tracking element 210. The display elements 203R, 203L are modulated by the maximum number possible over an angular region extending over the angular distance between the viewer's near and far fields, which is approximately 7 ° in total. Note that the viewpoints of can be modulated. Nevertheless, applying the centralized visual decompression method of the above embodiments, the method truncates the modulation basis coefficients in such a way as to adapt the angular perceptual visual acuity of the HVS as described above. And modulation, which effectively combines the fovea focused visual decompression and the compression gain of the foveated multi-view light modulator elements 203R, 203L of FIG. 5a. That is, in the example above, the foveated centralized visual decompression that achieves a medium compression gain factor of 32 times, and the foveated centralized multi-view light modulator element 203R of FIG. , 203L, the combined compression that can be achieved by the near-eye multi-view light field display system 200 reaches a compression gain factor of 96 times compared to a near-eye display system that achieves an equivalent viewing experience. It provides a near-eye light field display capability of 16 viewpoints each.

Multifocal Plane (Surface) Light Field-FIG. 6a illustrates an embodiment applying the visual defrosting method of the present invention in the context of a multifocal plane (surface) near-eye light field display. As shown in FIG. 6a, in this embodiment, the m-pixels and M-pixels of the light field modulators 203R, 203L are collimated and directionally modulated ray bundles (or light field anglelets) 610R, 610L. Designed to produce, they together extend angularly into the FOV of the near-eye lightfield display 200. In the present embodiment, the near-eye light field display 200 comprises a right side light field modulator 203R and a left side light field modulator 203L, each of which comprises a plurality of m-pixels and M-pixels, respectively. It is designed to generate a plurality of right light field anglet 610R and left light field anglet 610L pairs that address corresponding points in the retina of the viewer's right eye 580R and left eye 580L. (The Retina Corresponding Point is the point on the retina of the viewer's opposing eye whose sensory output is perceived by the viewer's visual cortex as a single point at a certain depth.) Right side light The pair of the right side light field anglet 610R and the left side light field anglet 610L generated by the light field modulator 203R and the left side light field modulator 203L, respectively, is a pair of these light field anglets in this specification. When 610R, 610L address a set of corresponding points on the retina of the viewer's right eye 580R and left eye 580L, it is said to be "visually corresponding". A pair of "visually corresponding" light field anglets 610R, 610L generated by the right light field modulator 203R and the left light field modulator 203L and relayed by the optical element 206 to the viewer's eyes 580R, 580L. The point within the FOV of the near-eye light field display 200 at which the points intersect is a virtual point of light (VPoL) within the light field modulated by the near-eye light field display system by the viewer's visual field. ) 620, which is perceived by both eyes. The binocular perceptual aspect of the viewer's HVS is that a visually viewed anglet flux image relayed by the optical element 206 onto the retinas of the viewer's eyes 580R, 580L provides a single viewed light spot. , To the virtual light spot (VPoL) 620 that is perceived at a depth corresponding to the corresponding eye separation distance of the viewer's eyes 580R, 580L. Thus, in this embodiment, the near-eye light field display 200 provides a “visually corresponding” light field anglet for the virtual light spot (VPoL) 620 that the viewer will perceive with both eyes within the FOV of the display. 610R and 610L are simultaneously modulated by the right side light field modulator 203R and the left side light field modulator 203L, respectively, to thereby modulate (or generate). The position of the virtual light spot (VPoL) 620 that the viewer perceives with the both eyes in the FOV of the near-eye light field display 200 is the right side light field that generated the “visually corresponding” light field anglet 610R, 610L. Determined by the (x, y) _R and (x, y) _L spatial (coordinate) positions of the m-pixels and / or M-pixels in the modulator 203R and the left light field modulator 203L. Therefore, the near-eye light field display 200 has (x, y) _R and (x, y) _R and (x, y) of m-pixels and / or M-pixels in its right side light field modulator 203R and left side light field modulator 203L. By addressing the _L space (coordinates) position, the virtual light spot (VPoL) 620 that the viewer perceives with both eyes can be modulated (generated) at any depth within the FOV of the near-eye light field display 200. In practice, using this VPoL 620 modulation method, the near-eye lightfield display 200 will "view" three-dimensional (3D) lightfield content within the FOV of the display that the viewer can focus on. “Correspondingly” pairs of light field anglets 610R, 610L are modulated by their right and left light field modulators 203R and 203L, respectively. The term "viewer focusable", in this context, allows a viewer of the near-eye lightfield display 200 to freely focus on an object (or content) within a modulated lightfield. Used to mean that they can be matched. This is an important feature of the near-eye lightfield display 200, which contributes significantly to reducing the above-mentioned VAC problems experienced by typical 3D displays.

  The inherent ability of HVS's depth-perceived vision eliminates the need for addressing all potential virtual light spots (VPoL) 620 within the FOV of near-eye lightfield display 200. The reason is that the binocular perception aspect of HVS on which binocular depth perception is achieved is achieved when viewing an object at a given eye separation distance (or position) from the viewer's eye, This is because an image is formed in a corresponding region (point) of the retina of the viewer's eye. The locus of all such positions (or eye separation distance) away from the viewer's eyes is known as the horopter surface. Combining the angular distribution of HVS visual acuity with its binocular aspect of perception produces a depth region surrounding the horopter surface, known as the phantom fusion region (or volume), which is perceived by the viewer throughout. Binocular depth perception is achieved even when the target object is not actually on the horopter surface. The binocular depth perceptual volume of the horopter surface, as extended by the associated phantom fusion area surrounding the horopter surface, causes the light field to be divided into individual sets of surfaces separated by the approximate size of the punom fusion area. Sampling with and (with some overlap, of course) suggests a way to ensure the continuity of binocular depth perception within a volume between multiple light field sampling surfaces. Empirical measurement (Hoffman, M .; Girshick, A.R .; Akeley, K. and Banks, M.S., Vergence-conformation affiliations vulnerabilities verbs (8) : 33, 1-30), the continuity of binocular depth perception is achieved when multiple 2D light-modulating surfaces are presented within the viewer's field of view, separated by approximately 0.6 diopters (D). It has been demonstrated that it can be achieved. Therefore, a set of horopter surfaces within the viewer's FOV, spaced apart by 0.6D, is for the viewer's HVS to achieve binocular perception in volume across such multiple horopter surfaces and associated phantom fusion regions. Will be enough. Here, a horopter surface separated by a distance necessary to achieve continuity of the viewer's binocular depth perception in an FOV extending from the viewer's near field to far field is referred to as a “standard horopter surface” Call.

In this embodiment, the near-eye light field is separated by a standard (ie, sufficient to achieve continuous volumetric binocular depth perception) spacing of 0.6D (holopter surface separation). The above method of sampling into a set of m-pixels and / or M-pixels (x, y) _R and (x) in the right side light field modulator 203R and the left side light field modulator 203L respectively. , Y) by defining a set of _L spatial positions using the virtual light spot (VPoL) 620 modulation method of the near-eye light field display 200 described in the previous embodiment, which is Generate a set of "visually corresponding" light field anglets, which are subsequently placed in a selected standard horopter surface set within the FOV of the display system 200. , Cause the perception in both eyes of a plurality of virtual point (VPoL) 620 by the viewer. Using the above method of modulating a standard set of horopter surfaces using the virtual light spot (VPoL) 620 modulation method described above, the near eye light field display 200 addresses the viewer's perception of the entire near eye light field. Can be specified. Thus, in practice, the method of this embodiment is proportional to the size (VPoL) of the selected horopter modulation surface relative to the size (VPoL) of the total light field that the near-eye light field display 200 is addressable. , Achieves a light field compression gain, which can be a sizeable compression gain, which is expected to be well over 100 times. Note that such compression gain is achieved by the virtual light spot (VPoL) 620 modulation capability of the near-eye lightfield display 200 in matching binocular perception of HVS and angular acuity.

FIG. 6b illustrates the near-eye light field holopter sampling and modulation method of the above embodiment. FIG. 6b shows a top view of the light field horopter surfaces 615, 618, 625, 630, 635, 640 relative to the position of the viewer's eye 610 from the viewer's near field (~ 30 cm) to the far field (~ 300 cm). Show regularly. As FIG. 6b shows, the first light field horopter surface 615 is at the near field distance of the viewer located 3.33D from the viewer and the remaining five light field horopter surfaces 618, 625, 630. , 635, 640 are located at consecutive 0.6D distances from the viewer's eye, ie, 2.73D, 2.13D, 1.53D, 0.93D, 0.33D. The six light field horopter surfaces 615, 618, 625, 630, 635, 640 shown in FIG. 6b each have a plurality of VPoL 620 modulated at a density (or resolution) corresponding to the depth and angular acuity of HVS. , For example, the density (spot size) of the modulated VPoL 620 on the first light field horopter surface 615 is 40 μm to match the spatial vision of the HVS at that distance and the remaining 5 light The field horopter surfaces 618, 625, 630, 635, 640 are continuously increased to match the spatial and angular vision distribution of the HVS. The plurality of VPoLs 620 forming the six light field horopter surfaces 615, 618, 625, 630, 635, 640 are the right side light field modulator 203R and the left side light field modulator 203L of the near-eye light field display 200, respectively. , Associated with a plurality of "visually corresponding" generated by a predetermined set of m-pixels and / or M-pixels at respective spatial positions (x, y) _R and (x, y) _L. Modulated by a pair of light field anglets 610R, 610L. (X, y) _R in the right light field modulator 203R and the left light field modulator 203L that modulate (generate) each of the six light field horopter surfaces 615, 618, 625, 630, 635, 640 and The (x, y) _L spatial position is calculated a priori and maintained by the visual decompression transform block 302, which constitutes each of the six light field horopter surfaces 615, 618, 625, 630, 635, 640. Yes, the corresponding VPoL 620 is addressed based on the lightfield image data 301 received by the visual decompression conversion block 302 as input from the embedded or external processor 102 or 107.

  Depth Foveated Concentrated Visual Defrost in Multifocal Plane Light Field-Right Side Light Field Modulator 203R and Left Side Light Field Modulator 203L of near-eye light field display system 200 may optionally have six light field horopters. All surfaces 615, 620, 625, 630, 635, 640 may be modulated simultaneously, but this is not required. This is because, at any particular moment, the viewer's eye focuses on a certain distance and, as mentioned above, the HVS's depth-perceived visual acuity is at its maximum near this point, and the depth from this point is Or, it is because it regularly decreases with the deflection of the angle. Thus, in this embodiment, the multifocal plane near-eye display system 200 of the present invention provides for the six light field horopter surfaces 615, 618, 625, 630, 635, 640 to be at the same time as the HVS vision at the viewer's focus. Visual compression gain is achieved using the multifocal surface light field modulation method of the present invention, which is modulated with a matching VPoL 620 density (resolution). Further, the above method of modulating a near-eye light field with a VPoL 620 that modulates a standard horopter surface 615, 618, 625, 630, 635, 640 as shown in FIG. 6b, and previous embodiments. In an embodiment incorporating both the foveated focused visual decompression method described in Section 1., in the near-eye display system 200, the sensed viewer focus provided by the eye and head tracking element 210 sensor is used. To determine (identify) the horopter surface that contributes most of the visual information in the vicinity of the point where the viewer's eye is in focus, and then apply the above-described foveal-focused visual defrosting method to produce six light exposures. The VPoL 620 that modulates the field horopter surface 615, 620, 625, 630, 635, 640 is placed in the vicinity of the above where the viewer's eye focuses. To compress in proportion to the contribution to the visuals information. In this embodiment, using the sensed viewer focus provided by the eye and head tracking element 210 sensor, less than 0.6 D (eye separation distance) from the position where the viewer's eye is in focus. Identify the light field horopter surface. This criterion defines a maximum of two of these standard light field horopter surfaces 615, 618, 625, 630, 635, 640 if the viewer is not focused on those surfaces. Will be identified (in this case only one of the horopter surfaces will be identified). As mentioned above, the viewer's HVS binocular fusion area actually fills the area of 0.6D between the standard light field horopter surfaces, so by the above criteria, It is ensured that the optical depth falls within the binocular fusion zone of at least one of the selected (identified) light field horopter surfaces. In this embodiment, the horopter surface identified using the selection criteria described above contributes most of the visual information in the vicinity of the point where the viewer's eye is in focus and fits, and thus, in FIG. The multifocal plane light modulator (display) elements 203R, 203L modulate the identified holopter surfaces by using the density of the VPoL 620 to match the HVS's visual acuity at the sensed depth of these surfaces, It also uses the maximum number of modulated basis coefficients with minimal truncation of word length to achieve the best visual perception, while contributing contributions near the point where the viewer's eye is in focus. The smaller remaining horopter surface was used with a smaller number of VPoL620s separated by a wider angular spacing, with proportionally fewer word lengths truncated. Using the modulation base coefficient, multifocal surface light modulator of FIG. 6a (display) elements 203R, is modulated by 203L. The net effect of the method of this embodiment is a three-dimensional foveated centralized visual defrosting action, in which the visual information near the point where the viewer's eye is in focus is the HVS of the focus. Modulated with the highest fidelity to match the perceived visual acuity, while the visual information in the surrounding area is at points farther (forward, backward and side) from the point where the viewer's eyes are in focus. , With a fidelity level adapted to the perceived visual acuity of the HVS which is proportionally reduced. The combined method of this embodiment is collectively referred to as multifocal plane light field depth foveated centralized visual thawing. It should be noted that the term "fovea focused" as used in the context of this embodiment is such that the display resolution is outward from the center of the fovea of the viewer's eye towards the peripheral region of the retina of the viewer's eye. , HVS is intended to be adapted to the depth perception visual acuity profile (distribution).

  It should be noted that in the above embodiment, the higher density VPoL 620 is shown in FIG. 6a for the central region of the fovea of the viewer's eye (402-404 in FIG. 4a) as shown by the eye and head tracking element 210. The display elements 203R, 203L of FIG. 6a are modulated by the elements 203R, 203L but extend over the angular distance between the viewer's near and far fields (which is approximately 7 ° in total). The maximum density VPoL 620 possible over the dynamic range can be modulated. Nevertheless, applying the depth foveated centralized visual decompression method of the above-described embodiment, the method modifies the modulation basis in such a way as to adapt the angular and depth perceptual visual acuity of the HVS as described above. Truncation and modulation of the coefficients is performed, which effectively combines the foveated visual decompression and the compression gain of the foveated multifocal surface light modulator (display) elements 203R, 203L of FIG. 6a. That is, in the example above, a foveated focused visual decompression that achieves a moderate compression gain factor of 32 times is reduced to a compression gain factor of approximately 3 times (this is a maximum of 2 out of 6 standard horopter surfaces). 6a, which achieves a density of VPoL620 of all six standard horopter surfaces may be centered in the fovea). , 203L, combined compression, which in this case can be achieved by the near-eye lightfield display system 200, achieves an equivalent viewing experience, near-eye display using a near-eye lightfield display with 6 focal plane performance. A gain factor of 96 is reached compared to the system.

FIG. 7 illustrates the generation of content for the multifocal plane near-eye lightfield display 200 of FIG. 6a. In this illustrative example, the scene is captured by camera 701 in three depth planes: a proximal plane, a midplane, and a distal plane. It should be noted that the more depth planes captured by the camera 701, the better the viewer's perception of depth in the multifocal surface light field near-eye display 200 of FIG. 6a. Preferably, the number of capture depth planes is such that the light field near-eye display 200 of FIG. 6a can modulate a focal plane (in the case of the embodiment described above, the six standard horopter surfaces 615, 618, 625 of FIG. 6b). 630, 635, 640). Although this example uses three capture planes to illustrate a further aspect of the invention, one of ordinary skill in the art, using the methods described herein, would use three capture planes of this illustrative example. A multifocal plane near-eye imaging (which means capture and display) system could be implemented that utilizes more planes than the depth plane that is captured. In this illustrative example, three objects, one object 702 near the capture camera and two other objects 703, 704 far from the camera, are placed in the content scene. For multifocal plane imaging systems, it is necessary to adjust the brightness of the object depending on the position of the object with respect to the (capture) depth layer. In the illustrative example of FIG. 7, this is done by depth filtering the intensity of the image content as indicated by filtering blocks 705, 706, 707 to match the intensity of the object in the image scene to its depth value. Is achieved. For example, the closest object 702 is illustrated with its full intensity in this particular layer 708 because it is entirely contained within the first depth layer, but is completely visible from the other two layers 706, 707. Will be removed. In the case of the intermediate object 703, it is located between the two depth layers (medium and distal), so to render the full intensity of the object 703 it is split into two layers 706, 707. To be done. However, since the perceived brightness of the object is the sum of all layers 711, the object is perceived at full brightness in the viewer's eye as a weighted sum of the brightness contributions from both depth planes 706, 707. become. To achieve a 3D perception of the scene, each depth layer 708, 709, 710 is displayed to the viewer at its corresponding depth, where the adjusted brightness corresponds to the depth of the objects in the scene. , This effectively evokes the viewer's depth cue, allowing the viewer to focus on the displayed content. The viewer will see a combination of all layers, which results in a reconstructed stereo image 711 with the appropriate focus cues for the viewer's HVS. By rendering the image content of the three capture planes of this illustrative example together with their relative depth information as described above, the colors and intensities of the image content of these planes are compared to the multifocal plane near-field of FIG. 6a. It is distributed (or mapped) on the multifocal plane of the eye display 200. The final result of this captured image rendering process is the color and brightness of the content of the input image 301 for each of the right and left light field modulators 203R and 203L of the near-eye light field display 200. A plurality of "visually corresponding" light field anglets 610R generated by respective sets of m-pixels and / or M-pixels in (x, y) _R and (x, y) _L spatial positions therein. , 610L pairs of color and intensity data into a dataset that specifies the data. Modulation of these color and brightness data sets by the right light field modulator 203R and left light field modulator 203L of the near-eye light field display 200, respectively, is rendered to the viewer as a set of modulated VPoL 620. It is able to perceive the content of the 3D image input and also freely focus on any of the displayed 3D objects 702, 703 or 704 of this scene. Note that, in the above illustrative example, only three capture planes are used, but even in this case, the near-eye light field display 200 of the present invention uses the above-described method of the present embodiment. Using the field display capability and using the VPoL620 modulation method described above, the input image data 301 is rendered to the six standard holopter surfaces of Figure 6b for display of the input image content.

The multi-focal plane depth filtering process shown in FIG. 7 effectively reduces the intensity of the content of the input image scene to the relevant input image depth in order to produce a perceived depth cue that is appropriate for the viewer's HVS. The process of assigning (or mapping) to the multifocal plane of the display 200 according to the information. In one embodiment of the present invention, the multi-focal plane near-eye lightfield display 200 of the present invention implements a local depth filtering process to provide the near-eye lightfield display 200 of FIG. In this case, there are six standard horopter surfaces located within the FOV of the viewer's near-field to far-field display as shown in Figure 6b). it can. FIG. 8 illustrates a multifocal plane depth filtering method 825 of this embodiment, which causes the layer splitter 802 to process the image input 301 and associated depth map 801 to generate an image depth plane or layer, Corresponds to the capture depth plane. Depth filtering 803 is then performed on the generated content of each layer to map the input image 301 and associated input depth map 602 onto the displayed multifocal plane image. Next, the image rendering block 804 uses the generated multifocal plane image to modulate the VPoL 620 of the multifocal plane to the viewer of the display, the right side light field modulator 203R and the right eye field modulator 203R of the near eye light field display 200. Within each of the left side light field modulators 203L, a plurality of "visions" produced by respective sets of m-pixels and / or M-pixels at (x, y) _R and (x, y) _L spatial locations. And "corresponding" light field anglets 610R, 610L pairs of color and intensity values.

  In another embodiment, the display images for the standard lightfield horopter surfaces 615, 620, 625, 630, 635, 640 of the near-eye lightfield display 200 of the above-described embodiments are referenced to the content of the captured scene. Generated from an input image 301, which is composed of a compressed set of elemental images or holographic elements (Hogel) (see US Published Patent 2015/0201176). In this embodiment, the elemental images or hogels of the scene captured by the lightfield camera are first processed to (standard designation of the standard lightfield horopter multifocal surfaces 615, 620, 625, 630, 635, 640. The smallest number of captured elemental images or hogel subsets that contribute to the best or full representation of the image content at the specified depth. This identified subset of elemental images or hogels is referred to herein as a reference hogel. Identified, containing the image content of standard multifocal surfaces 615, 618, 625, 630, 635, 640 for the data size of the total number of elemental images or hogels captured by the source light field camera of the scene. The above reference hogel data size exhibits a compression gain that is inversely proportional to the result of dividing the data size of the identified reference hogel subset by the total number of captured elemental images or hogels, which is greater than 40 times. The compression gain may be reached. Thus, in this embodiment, the captured light field datasets are compressed into data sets representing individual sets of multifocal surfaces of the near-eye light field display 200, and by implementing this, the embodiments described above. Of standard light field holopter multifocal surfaces 615, 618, 625, 630, 635, 640 identified by the method described above to achieve compression gain by adapting aspects of the viewer's HVS for depth perception. A compression gain is realized, reflected as being a compressed representation of the field.

Compressed Rendering—In another embodiment, shown in FIG. 9, “compressed rendering” (US Published Application 2015/0201176) is referred to as the compressed light field dataset of the reference hogel of the above embodiment. , For the modulation of the light field image at the standard light field holopter multifocal surface 615, 618, 625, 630, 635, 640 by performing directly on the received image input 805. Images to be displayed are extracted by the right side light field modulator 203R and the left side light field modulator 203L of the focal plane near-eye light field display 200. FIG. 9 illustrates a compressed rendering process 806 of this embodiment, which processes input light field data 805 including a reference hogel compressed light field dataset to produce a multifocal near-eye light field display 200. The inputs to the right side light field modulator 203R and the left side light field modulator 203L are generated. In the compression rendering process 806 of FIG. 9, the received compressed input image 805, including the reference hogel light field dataset of the embodiment described above, is first rendered to a standard light field horopter multifocal surface 615. , 620, 625, 630, 635, 640, the light irradiation field image is extracted. In the first step 810 of the compression rendering process 806, the reference hogel image and associated depth and texture data including the light input 805 is used to generate a standard light field holopter multifocal surface 615, 618, 625, The color and luminance value of VPoL of the near-eye light irradiation field that constitutes each of 630, 635, and 640 is synthesized. Since the reference hogel was selected a priori based on the depth information of the standard light field horopter multifocal surfaces 615, 618, 625, 630, 635, 640, the VPoL synthesis process 810 was compressed. Minimal processing throughput and memory are required to extract the VPoL color and intensity values of the near-eye light field from the reference hogel input data 805. Further, as shown in FIG. 9, the viewer's gaze direction and depth of focus sensed by the eye and head tracking element 210 are used in the VPoL compositing process 810 to view the viewer's sensed gaze direction and depth of focus. Render the VPoL value based on the HVS visual acuity distribution profile of the person. Associated with each of the synthesized near-eye light field VPoL values is the direction of the visually corresponding anglet pair, and within each of the right and left light field modulators 203 of the near-eye light field display 200. (X, y) _R and (x, y) _L spatial position coordinates of the pair of anglets. Next, the color and intensity values of the visually corresponding anglet pairs associated with each of the extracted near-eye light field VPoLs are processed by the anglet synthesis process 815 to the right light field modulator 203R and left light field. Each of the field modulators 203L is mapped (converted) into (x, y) _R and (x, y) _L space position coordinates. Depending on the viewer's line-of-sight direction sensed by the head and eye tracking elements 210, the depth foveated centralized visual compression block 820 utilizes the method described in the above embodiments to utilize the right side light field modulator. 203R and the left side light field modulator 203L, the generated color and luminance values relating to the (x, y) _R and (x, y) _L spatial position coordinates in the viewer's HVS visual acuity distribution, respectively. Compress based on. In essence, this embodiment has three compression gains of the above-described embodiments, namely: (1) the smallest reference hogel of light field data that completely composes a standard light field multifocal surface. Gain related to compression into a set of (2) all-light fields to the set of VPoLs that make up each standard light field multifocal surface; and (3) the viewer's Fig. 7 is a combination of the gains associated with depth foveal focusing of the modulated VPoL for HVS angular, color and depth vision adaptation. The first of these compression gains significantly reduces the interface bandwidth of the near-eye display system 200; the second of these compression gains is due to the VPoL and the corresponding anglet generated. It significantly reduces the computational resources required; the third of these compression gains significantly reduces the interface bandwidth of the near eye display light field modulators 203R, 203L. It should be noted that the effect of these compression gains is on the near-eye display light field modulators 203R, 203L, which allows the compressed input to be displayed directly without the need to decompress first as is currently done in prior art display systems. It is further enhanced by the compressed display capability.

  By the above description of the embodiments, an image compression method for a near-eye display, which reduces an input bandwidth and a system processing resource, has been presented. Higher-order basis modulation, dynamic color gamut, light field depth sampling, and image data word length truncation and quantization for angular, color and deep vision of the human visual system, with the use of compressed input displays. Combined, it enables a high-fidelity visual experience in near-eye display systems suitable for mobile applications with significantly reduced input interface bandwidth and processing resources.

  It will be apparent to those skilled in the art that various modifications and variations can be applied to these embodiments of the present invention without departing from the scope of the invention as defined in and by the appended claims. Will be understood by. It is to be understood that the above-described examples of the invention are merely exemplary, and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. For example, various possible combinations of the disclosed embodiments can be used to achieve additional compression gain in near-eye display designs not specifically described in the illustrative examples above. Therefore, the disclosed embodiments, either individually or in any combination, should not be considered limiting in any sense. The scope of the invention is indicated by the appended claims rather than the foregoing description, and all variations that come within the meaning and range of equivalency of the appended claims are intended to be included within the scope of the invention. ing.

Claims (49)

A method of forming a near eye display, comprising:
Optically coupling at least one image display element to the eyes of a near-eye viewing viewer having at least one corresponding optical element; and at least one image display element,
An image processing element by electrically coupling the image processor element to the encoder element and coupling the encoder element to the image display element by embedding the image processing element and the encoder element in a near-eye display system near an observer's eye. And an encoder element may be selected, or the image processor element and the encoder element may be located away from a viewer's eye and the encoder element may be coupled to the near-eye display system by a wireless or wired connection.
Optically coupling at least one eye and head tracking element in the near-eye display to sense the line-of-sight and focal length of the near-eye viewer, and sensing the line-of-sight and focal length of the near-eye viewer. Steps,
Coupling the outputs of the eye and head tracking elements to the image processor and encoder element; coupling the outputs of the image processor and head tracking element to the image processor and encoder element;
A method of providing image data to an encoder element, the encoder element providing compressed image data to a near-eye display element.   The image display element directly displays the image content of the compressed image data received from the encoder element without first decompressing the compressed image data. The method described.   The method of claim, wherein the encoder compresses the image data into a compressed image data format, wherein the image display element includes a compressed image data received from the encoder element without first decompressing the compressed image data. The method of claim 1, wherein the image content in the different image data format is displayed directly.   The compressed image data is a macro basis modulation, which is an expansion coefficient of either discrete Walsh, discrete wavelet or discrete cosine image transform, formatted with reference to a set of upper macros containing one t. The method of claim 3, comprising a plurality of nxn pixels having coefficients.   The image display element modulates the compressed image data at a sub-frame rate that allows the human visual system of the near-eye display system to integrate and perceive the compressed image data directly with the decompressed image. Method according to claim 3, characterized in that   The compressed image data format is referred to by a color gamut of an image frame or a subframe, and the encoder element embeds the color gamut of the image frame or a subframe in the compressed image data format to generate an image display element. Is a compressed image data that dynamically adjusts its color gamut at the frame or sub-frame rate of the compressed image data format and modulates the compressed image data directly into the color gamut of the image frame or sub-frame. The method of claim 3, embedded in a format. The encoder element is a visual expansion conversion element that extracts a base modulation coefficient from the image data, and a visual expansion conversion element that extracts the base modulation coefficient from the image data,
First, a quantizer element that truncates the extracted base modulation coefficient into a subset of the extracted modulation coefficient based on a coefficient set truncation criterion, the quantizing element further selects a selected subset of the extracted modulation coefficient. , A word length of the extracted subset of the base modulation coefficients to be quantized using a word length shorter than t is determined based on a quantization criterion of the coefficient set, and based on the quantization criterion of the coefficient set, Determining the word length of the extracted subset of the base modulation coefficients,
Temporally multiplex the truncated and quantized subsets of the extracted base modulation coefficients and transmit the multiplexed truncated and quantized subsets of the extracted base modulation coefficients as compressed image data 5. The method of claim 4, including a run-length encoder element that:   8. The coefficient set truncation criterion discards extracted basis modulation coefficients associated with image transforms having a temporal response at frequencies above the temporal perceptual visual acuity limit of the near-eye display system viewer's visual system. The method described in.   8. The method of claim 7, wherein the coefficient set quantisation criterion selects successively shorter word lengths for the image transform having a higher frequency temporal response.   The coefficient set quantization criterion further comprises a word length proportional to a frame or frame region gamut size for the image display element standard gamut size, Method according to claim 7, characterized in that it is chosen to be small in size and the word length used to represent the color coordinates of the selected image transform is small. The encoder element is
Visual expansion conversion element for extracting a base modulation coefficient for a set of (nxn) A high-order macro is extracted from the compressed image data based on the line-of-sight direction of the eye, and the observer detected by the head tracking element. Extract higher order macros from said compressed image data based on gaze direction; eye and head tracking elements for first truncating the set of extracted basis modulation coefficients to a subset of basis modulation coefficients based on the coefficients A foveated quantizer element that utilizes the viewer's line-of-sight direction sensed by, and by setting a truncation criterion, the foveated quantizer element further includes a subset of the base modulation coefficients. Is quantized using the word length, but is shorter than the word length of the extracted subset of the base modulation coefficients based on the quantization criterion of the coefficient set. A run that temporally multiplexes the truncated and quantized subsets of the modulation coefficients and combines the multiplexed truncated and quantized subsets of the base modulation coefficients as compressed image data into an image display element. The method of claim 4, further comprising a length encoder element.   A basis modulation coefficient set truncation criterion discards extracted basis modulation coefficients associated with a base modulation coefficient having a temporal response at a frequency higher than the temporal perceptual visual acuity limit of the near-eye display system viewer's visual system. Item 11. The method according to Item 11.   The base modulation coefficient set truncation criterion selects more extracted base modulation coefficients for the central region of the viewer's eye field of view and is determined by the viewer's gaze direction sensed by the gaze and head tracking elements, The method according to claim 11, wherein the method is further configured to generate successively fewer base modulation coefficients towards a peripheral region of the visual field of the viewer's eyes.   The basis modulation coefficient set quantization criterion is a field of view of an observer's line of sight, which selects successively shorter word lengths and longer word lengths t for basis modulation coefficients having a higher frequency time response. For the quantization of the base modulation coefficient for the central region of the, continuous for the quantization of the base modulation coefficient toward the surrounding region of the eye determined by the eye gaze direction sensed by the eye and head tracking elements. 12. The method according to claim 11, wherein a short word length is selected.   The basis modulation coefficient set truncation criterion is characterized by selecting the upper macro of the compressed image data for the central region of the visual field of the viewer's eyes, and the visual direction of the viewer sensed by the gaze and head tracking elements. 12. The submacro for the peripheral region of the observer's eye field of view is sequentially selected as determined by the observer's gaze direction sensed by the gaze and head tracking elements, as determined by Method.   The base modulation coefficient set truncation criterion is used to represent the base modulation coefficient based on the display gamut where consecutively short word lengths depend on the viewer's human visual system color vision profile with respect to the viewer's gaze direction. 12. The method of claim 11, wherein the near eye display system viewer selects a word length that depends on the color vision profile of the human visual system.   The method of claim 1 using a reflector and beam splitter optics assembly, a free-standing optical wedge or a waveguide optics. Optically coupling at least one light field image display element to each of the eyes of a near-eye light field display viewer having corresponding optical elements; and at least one light field image display element,
Image by electrically coupling an image processor element to an encoder element and coupling the encoder element to an image display element, and by embedding the image processing device and the encoder element in a near-eye light field display system in the vicinity of an observer's eye. Providing a method of controlling a processing device, or arranging an image processor and an encoder element away from a viewer's eye and coupling the encoder element to a near-eye optical field display system by a wireless or wired connection;
Optically coupling at least one eye and head tracking element in the near-eye optical field display system for sensing each of the near-eye viewing viewer's line-of-sight direction and focal length, and the near-eye viewing viewer's line-of-sight direction. And sensing each of the focal lengths,
Coupling the outputs of the eye and head tracking elements to the image processor and encoder element; coupling the outputs of the image processor and head tracking element to the image processor and encoder element;
A method of forming a near-eye optical field display system in which an image processing element provides optical field image data to an encoder element, the encoder element providing compressed optical field image data to the optical field image display element.   The light field image display element modulates each aspect of the near-eye light field viewer's human visual system with a sample of the light field displayed to a near-eye light field display system viewer, either as multiple views or as multiple views. Is configured to display each physical pixel of the right and left light field image display elements of a near-eye light field display system sampling using multiple (mxm) groups as a focal plane sample of The method according to claim 18.   The light field samples are modulated by the right and left light field image display elements of the near-eye light field display system, each collimated and directionally coupled to a corresponding optical element via a set of micro-optical elements. Each micro-optical element associated with each of the physical pixels is a micro-optic within each group of multiple (mxm) physical pixels of the right and left light field image display elements. 20. The method of claim 19, including an optical aperture for each set of elements.   Each set of micro-optical elements associated with each of the physical pixels and each group of physical pixels of the light field image display element collimates and directionally modulates the anglet at time t. 21. The method of claim 20, wherein the higher angular density of the angle within the central region of the optical aperture of the light field image display element is greater than the angular density of the angle within the peripheral region of the light field image display element. .   The distribution of angular densities of the angle from the center of the light field image display element to the peripheral region is proportional to the angular distribution of the visual acuity of the human visual system of the viewer and is optically coupled onto the peripheral region of the retina of the observer eye. 22. The method of claim 21, wherein the method allows for optically coupled onto a central retinal region of an observer's eye with a systematically reduced angular density of an angle.   An observer between the near-field and the far-field of the observer of the near-eye optical field display system, characterized in that the central region of the optical opening of the bright-field image display element comprises the maximum density anglet. 19. The method of claim 18, wherein the angular width is wide enough to accommodate the eye movements of the.   An observer between the near-field and the far-field of the observer of the near-eye optical field display system, characterized in that the central region of the optical opening of the bright-field image display element comprises the maximum density anglet. The dimension of the group of physical pixels in the central optical region of the light field image display element, which is wide enough to accommodate the eye movement of the eye, is coupled to the eye of the observer via an optical element. 20. The method of claim 19, projecting a spot size that corresponds to the average spatial visual acuity of the central region of the retina of the observer eye.   The brightfield image display element modulates a higher number of views on a fovea region of a viewer and a systematically smaller number of views on a peripheral region of the viewer's field of view, 19. The method according to claim 18, whereby the angular velocity and depth perception of the human visual system of the viewer can be matched.   The optical field image display element directly displays the image content of the compressed image data received from the encoder element without first decompressing the compressed image data, and the encoder element includes the viewing element. Viewing based on the perceived focus position of the viewer provided by the eye and the head tracking element, characterized in that it provides compressed image data in the vicinity of the point where the human eye is in focus. At the perceived point of the viewer's focus, the visual information in the surrounding area is modulated with the highest fidelity to match the perceptual visual acuity of the viewer's human visual system, the visual information of the surrounding area By providing a near-eye optical modulation at a point away from where it is, with a fidelity level consistent with the proportionally lower perceptual visual acuity of the human visual system. To achieve over field display system, comprising providing a visual stretching function is depth 窩化 achieve a three-dimensional depth foveal vision compression by the light field image display element, the method according to claim 19.   The near-eye light field display system modulates a bright field that is focusable to an observer by modulating a pair of visually corresponding anglets from the left and right eye light field image display elements perceived at t. Virtual field of light in the field of view of the light field image display element in the field of view at a given depth determined by the spatial coordinates of the physical pixel groups of the left and right lights characterized by 20. The method of claim 19, wherein the field image display elements that provide the human visual system as generated generated visually corresponding pairs of anglets.   The near-eye light field display system presents a set of multifocal plane samples to an observer, whereby the multifocal plane is at t from the near field depth of the observer to the far field of the observer. 19. The method of claim 18, wherein a set of standard horopter surfaces extending to depth are nominally separated by 0.6 degrees. The near-eye light field display system modulates a bright field that is focusable to an observer by modulating a pair of visually corresponding anglets from the left and right eye light field image display elements perceived at t. A light field image in the field of view at a given depth determined by the spatial coordinates of the left and right physical pixel groups characterized by the characterizing method'the human visual system as a virtual point of light in the field of view Optical field image display element providing a visually generated pair
Corresponding to the corresponding angle, the near-eye light field display system presents the observer with a set of multifocal plane samples, whereby the multifocal plane is taken from the observer's near-field depth of one t. A set of standard horopter surfaces extending to the far depth of field, characterized in that the standard horopter surfaces are nominally separated by (0), the diopter is of magnitude t. Near-eye light field display system that modulates the canonical horopter plane at the virtual point of light of the selected canonical horopter plane using the virtual point of light that achieves proportional optical field modulation compression gain. 20. The method according to claim 19, for the magnitude at the visible point of light of the total light field addressable by the display system.   The near-eye light field display system modulates a bright field that can be focused on an observer by modulating a pair of visually corresponding anglelets from the display elements of the left and right eyes perceived at t. The viewer's human visual system as a virtual point of light in the field of view at a given depth determined by the spatial coordinates of the physical pixel groups of the left and right light field images. A display element for producing a pair of visually corresponding angles for display in a field of view, a near-eye optical field display system presents to an observer a set of multifocal surface samples in which multifocal planes are formed. A set of standard horopter planes extending from the observer's near field depth to the observer's far field depth, where the standard horopter plane is nominally divided by (0). The density of modulated virtual points of light comprising each of the standard horopter surfaces fitted to one t, characterized in that the acute angle of depth and angle of the human visual system of the viewer is 20. Method according to claim 19, characterized in that it is the corresponding distance of a standard horopter surface from. The near-eye light field display system modulates a bright field focusable to an observer by modulating a pair of visually corresponding anglets from the left and right sidelight field image display elements perceived at t. Optical field image display element in the field of view at a given depth determined by the spatial coordinates of the left and right physical pixel groups characterized in that the human visual system as a virtual point of light in the field Optical field image display element providing a visually generated pair
Corresponding to the corresponding angle, the near-eye light field display system presents the observer with a set of multifocal plane samples, whereby the multifocal plane is taken from the observer's near-field depth of one t. Using a virtual point of light, a set of standard horopter surfaces extending to the far field depth, characterized in that the standard horopter surfaces are nominally separated by (0). Near-eye optical field display system for modulating a canonical horopter plane, light proportional to the size of the virtual point of the selected canonical horopter plane for the magnitude of v of the total optical field addressable by the near-eye light field display. Achieving field modulation compression gain and achieving both combined optical field modulation gain and visual compression gain. The method according to claim 26.   The compressed optical field image data is a basis modulation coefficient of either discrete Walsh, discrete wavelet or discrete cosine image transform, formatted with reference to a set of upper macros containing one t, macro Less than 0.6 degrees from where the observer's eyes are in focus, using the sensed focus position of the observer provided by including a plurality of mxm pixels having a modulation basis modulation coefficient of At a perceived depth of the identified canonical horopter surface, which is configured to identify the canonical horopter surface at the highest visual using vpos density that matches the visual acuity of the human visual system of the viewer. Modulate the identified canonical Hopter surface to achieve perception and use the smallest word-length truncation that involves using the highest number of criteria. The modulation factor is higher, spaced by a wider angular pitch that is modulated with the rest of the canonical hopter surface, which has a small contribution near the point where the viewer's eye is in focus. 27. The method of claim 26, wherein depth foveal visual compression can be incorporated by using a proportionally small number of basis modulation coefficients in word length truncation.   To allow the user's human visual system to perceive the captured depth of the displayed content, a standard horopter surface is used to modulate the image content incorporating the corresponding depth cues. 29. The method of claim 28, further comprising further performing local depth filtering to generate all sets. The brightfield image data comprises a reference elemental image of the captured scene content or a compressed set of hogels that identifies the most captured elemental images or a subset of hogels, the method, a canonical brightfield horopter plane. Fully or fully representing the image content at the depth of the near-eye light field display system, wherein the near-eye light field display system comprises the canonical light field horopter surface from the set of compressed reference hogels of the captured scene. Characterized by rendering a display image about
Content that identifies a minimal number of captured hogel subsets that represent most or sufficient image content at the depth of the canonical brightfield horopter surface, compression gain inversely proportional to the data size of the identified subset of reference hogels, 29. The method of claim 28, inversely proportional to the total number of captured elemental images or hogels.   Characterized by applying compressed rendering directly to a set of compressed reference hogels to extract the image content displayed by the left and right side image display elements to modulate the displayed image in the canonical horopter plane. 35. The method of claim 34, wherein   19. The method of claim 18, using a reflector and beam splitter optics assembly, a free-standing optical wedge or a waveguide optics. At least one image display element for optically coupling to at least one eye having at least one corresponding optical element, and at least one image display element having at least one corresponding optical element;
The image display element is electrically coupled to receive image data from the encoder element, the encoder element is electrically coupled to receive image input data from the image processor element,
The encoder and processor are either embedded within the near-eye display system near the observer's eye or located away from the viewer's eye and wirelessly or wired to the near-eye display system assembly. Connected by connection,
At least one eye and head tracking element optically coupled to sense line-of-sight direction and focal length of the near-eye viewing viewer;
A near-eye display system comprising an eye and head tracking element electrically coupled to transfer detected data to an image processor and encoder element.   38. The system of claim 37, wherein the image display element receives compressed image data from the encoder element and displays the image directly without first decompressing the image data. The encoder element comprises a visual expansion transform element for extracting a base modulation coefficient for a set of higher order display macros containing (nxn) multiplicity, and the pixels having the modulation coefficient of the display macro are An expansion coefficient of either discrete Walsh, discrete wavelet or discrete cosine image transform from the input image data,
A quantization element that first truncates the extracted base modulation coefficient into a subset of base modulation coefficients based on a base modulation coefficient set truncation criterion;
The quantizing element further quantizes the selected subset of the base modulation coefficients using a word length shorter than the time t. Base modulation coefficient set A word of the extracted subset of the base modulation coefficients based on a quantization criterion. Determining a length, and determining a word length of the extracted subset of basis modulation coefficients based on a basis modulation coefficient set quantization criterion,
38. A run length encoder element that temporally multiplexes the truncated quantized subset of the bottom modulation coefficient and includes a run length encoder element that transmits the multiplexed truncated quantized subset of the base modulation coefficient to the image display element. The system described in. The encoder element further comprises
A visual expansion transform element for extracting a base modulation coefficient for a set of (nxn) th macros from the input image data based on the viewer's gaze direction sensed by the notation and head tracking elements;
The gaze direction of the viewer sensed by the eye and head tracking elements to first truncate the emitted set of basis modulation coefficients to a selected subset of basis modulation coefficients based on the basis modulation coefficient set truncation criteria. using,
A sub-elemental element further quantizes the selected subset of modulation values using a word length shorter than t, and determines the word length of the extracted subset of the base modulation coefficients based on a basis modulation coefficient set quantization criterion. Determining the word length of the extracted subset of basis modulation coefficients based on the basis modulation coefficient set quantization criterion.
38. A run length encoder element for time division multiplexing, time division multiplexing a truncated quantized subset of base modulation coefficients, and then combining the multiplexed data set with an image display element. The system described in. At least one image display element for optically coupling to an eye of a near eye display system having at least one corresponding optical element; an encoder element electrically coupled to receive image input data from an image processing element An image display element electrically coupled to receive image data from
The encoder and processor are either embedded within the near-eye display system near the observer's eye or located away from the viewer's eye and wirelessly or wired to the near-eye display system assembly. Connected by connection,
At least one eye and head tracking element optically coupled to sense each of the gaze direction and focal length of the near-eye optical field viewer;
A near-eye optical field display system comprising an eye and head tracking element electrically coupled to transfer detected data to an image processor and encoder element.   42. The system of claim 41, wherein the number of image display elements is a left and right image display element and the number of optical elements is two.   The image display element modulates a plurality of samples of a light field displayed to a viewer to respective sides of the viewer's human visual system, as a plurality of views or as a plurality of focal plane samples. (Mxm) groups for sampling the physical pixels of each of the image display elements to the right of the near-eye light field display system and to the left of the near-eye light field display system. 43. The system of claim 42, characterized by:   The optical field samples modulated by the right and left image display elements of the near-eye optical field display system are collimated, directionally modulated luminous fluxes or optical fields anglet, respectively, where t is t. Controlling a physical pixel of a plurality of (mxm) image display elements coupled onto a corresponding optical element via a set of micro-optical elements associated with each pixel forming a physical aperture. 44. The system of claim 43, comprising an image display device that does. A set of the micro-optical elements associated with each of the plurality of pixels and the plurality of groups,
The (mxm) physical pixels of the image display element are within range of collimating and directionally modulating the light field anglet with a higher angular density of the light field anglet than the higher t. 45. The system of claim 44, having an optical aperture that is greater than the angular density of the light field anglet in the peripheral region of the display element.   The image display element modulates a higher number of views on the viewer's foveal region and produces a systematically low number of views on a peripheral region of the viewer's field of view, thereby 43. The system according to claim 42, characterized in that it is adapted to the angular velocity and depth perception of the human visual system. The encoder further comprises a visual stretch transform element for extracting a base modulation coefficient for a set of (nxn), and from the input image data, the gaze direction of the viewer sensed by the eye and the head tracking element. On the basis of,
In order to first truncate the set of extracted base modulation coefficients to a selected subset of the base modulation coefficients based on the base modulation coefficient set truncation criterion,
The quantisation element further quantizes the selected subset of modulation values using a word length shorter than t. Base Modulation Coefficient Set Determines the word length of the extracted subset of the base modulation coefficients based on a quantization criterion. And determining the word length of the extracted subset of basis modulation coefficients based on the basis modulation coefficient set quantization criterion.
A run length coder element for time division multiplexing the time quantized subset of the truncated quantized subset of base modulation coefficients and then combining the multiplexed data set to an image display element;
43. The system of claim 42, which provides three-dimensional foveal visual compression based on where the viewer's eyes are in focus.   The image display element is a system characterized by presenting a set of multifocal planes to an observer, the multifocal plane extending from the observer's near-field to far-field depth, and nominally approx. 43. The system of claim 42, comprising a set of canonical horopter surfaces separated by 0.6 diopters.   43. The system of claim 42, wherein each optical element comprises a reflector and beam splitter optical assembly, a free-form optical wedge or a waveguide optical element.