Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
  • G09G5/39—Control of the bit-mapped memory
  • G09G5/391—Resolution modifying circuits, e.g. variable screen formats
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
  • A61B3/02—Subjective types, i.e. testing apparatus requiring the active assistance of the patient
  • A61B3/024—Subjective types, i.e. testing apparatus requiring the active assistance of the patient for determining the visual field, e.g. perimeter types
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
  • G02B27/0172—Head mounted characterised by optical features
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/013—Eye tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T15/00—Three-dimensional [3D] image rendering
  • G06T15/10—Geometric effects
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0101—Head-up displays characterised by optical features
  • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0101—Head-up displays characterised by optical features
  • G02B2027/0147—Head-up displays characterised by optical features comprising a device modifying the resolution of the displayed image
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2320/00—Control of display operating conditions
  • G09G2320/06—Adjustment of display parameters
  • G09G2320/0686—Adjustment of display parameters with two or more screen areas displaying information with different brightness or colours
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/04—Changes in size, position or resolution of an image
  • G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/366—Image reproducers using viewer tracking
  • H04N13/378—Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/366—Image reproducers using viewer tracking
  • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Definitions

• FR Foveated rendering
• VR virtual reality
• AR augmented reality
• Some FR techniques involve tracking a viewer’s eye gaze in real time using an eye gaze tracker integrated with a VR/AR headset. For a satisfactory viewer experience with FR, eye gaze tracking needs to have sufficiently high accuracy, fast speed, and low latency, which can be difficult to achieve. Some FR techniques do not use eye gaze tracking, and instead use a fixed focal point. Such FR techniques are referred to as fixed FR. For example, assuming that a viewer looks at the center of a display, the field of view (FOV) of the display can be divided into a central zone with maximum resolution, and several peripheral zones with reduced resolutions. Since the viewer may not always look at the center of the display, the viewer experience may be compromised. Other techniques such as content-based FR also do not require eye gaze tracking, but may require heavy computational resources.
• a method of generating foveated rendering using temporal multiplexing includes generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method further includes generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method further includes multiplexing the first spatial profile and the second spatial profile temporally in a sequence of frames, so that a viewer perceives images rendered in a region of the first foveated zone that does not overlap with the second foveated zone and/or in a region of the second foveated zone that does not overlap with the first foveated zone as rendered at the first pixel resolution.
• a method of generating foveated rendering using binocular multiplexing includes generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method further includes generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method further includes multiplexing the first spatial profile and the second spatial profile for a left eye and a right eye of a viewer, respectively, so that the viewer perceives images rendered in a region of the first foveated zone that does not overlap with the second foveated zone and/or in a region of the second foveated zone that does not overlap with the first foveated zone as rendered at the first pixel resolution.
• a method of generating foveated rendering using a combination of temporal multiplexing and binocular multiplexing includes generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method further includes generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method further includes generating a third spatial profile for the FOV by dividing the FOV into a third foveated zone and a third peripheral zone.
• the third foveated zone is spatially offset from the first foveated zone.
• the third foveated zone will be rendered at the first pixel resolution, and the third peripheral zone will be rendered at the second pixel resolution.
• the method further includes generating a fourth spatial profile for the FOV by dividing the FOV into a fourth foveated zone and a fourth peripheral zone.
• the fourth foveated zone is spatially offset from the third foveated zone.
• the fourth foveated zone will be rendered at the first pixel resolution, and the fourth peripheral zone will be rendered at the second pixel resolution.
• the method further includes multiplexing the first spatial profile and the second spatial profile for a left eye and a right eye of a viewer, respectively, in odd frames; and multiplexing the third spatial profile and the fourth spatial profile for a left eye and a right eye of a viewer, respectively, in even frames.
• a method of realizing video pipeline implementation of dynamically multiplexed foveated rendering includes rendering a foveated image that includes a foveated zone and a peripheral zone, wherein the foveated zone has a first set of image data and is rendered at a first pixel resolution and the peripheral zone has a second set of image data and is rendered at a second pixel resolution lower than the first pixel resolution.
• the method further includes packing the first set of image data into a first image block and packing the second set of image data into a second image block.
• the method also includes generating a control packet that includes rendering information associated with the foveated image, concatenating the control packet with the first image block and the second image block to form a frame, and transmitting the frame to a display unit.
• the control packet is parsed from the transmitted frame and is decoded to obtain the rendering information. Finally, a display image will be rendered according to the decoded rendering information.
• a method of realizing video pipeline implementation of dynamically multiplexed foveated rendering including time warp comprises rendering a foveated image that includes a foveated zone and a peripheral zone, wherein the foveated zone has a first set of image data and is rendered at a first pixel resolution and the peripheral zone has a second set of image data and is rendered at a second pixel resolution lower than the first pixel resolution and generating a control packet that includes rendering information associated with the foveated image.
• the method further includes time warping the foveated image for movement in viewer's position to form a time warped image and transmitting the time warped image and the control packet to a video processor.
• the method further includes remapping the time warped image into a foveated region packed data block and a low resolution region packed data block, concatenating the control packet with the foveated region packed data block and the low resolution region packed data block to form a frame, and transmitting the frame to a display unit.
• the control packet will be parsed from the frame and decoded to obtain the rendering information.
• the method also includes projecting a display image rendered according to the decoded rendering information.
• a method of realizing video pipeline implementation of binocularly multiplexed foveated rendering including time warp comprises rendering a first foveated image for the left eye that includes a first foveated zone and a first peripheral zone, wherein the first foveated zone has a first set of image data and is rendered at a first pixel resolution and the peripheral zone has a second set of image data and is rendered at a second pixel resolution lower than the first pixel resolution and rendering a second foveated image for the right eye that includes a second foveated zone and a second peripheral zone, wherein the second foveated zone has a third set of image data and is rendered at a third pixel resolution and the peripheral zone has a fourth set of image data and is rendered at a fourth pixel resolution lower than the third pixel resolution.
• the method further includes generating a control packet that includes rendering information associated with the first foveated image and the second foveated image, time warping the first foveated image and the second foveated image for movement in a viewer's position to form a first time warped image and a second time warped image, compressing the first time warped image and the second time warped image to form a first compressed image and a second compressed image, and transmitting the first compressed image, the second compressed image and the control packet to a video processor.
• the method further includes decompressing the first compressed image to a first recovered foveated image, decompressing the second compressed image to a second recovered foveated image, performing a second time warp on the first recovered foveated image and the second recovered foveated image based on the latest viewer's pose data, remapping the first recovered foveated image into a first set of three separate color channels to form a first channeled image, packing the first channeled image with the control packet to form a first frame, and transmitting the first frame to a first display for the left eye, wherein the first display for the left eye parses the control packet from the first frame, decodes the control packet to obtain the rendering information of each of the three separate color channels for the first frame, and saves the rendering information of the first frame in a memory of the first display.
• the method includes remapping the second recovered foveated image into a second set of three separate color channels to form a second channeled image, packing the second channeled image with the control packet to form a second frame and transmitting the second frame to a second display for the right eye, wherein the second display for the right eye parses the control packet from the first frame, decodes the control packet to obtain the rendering information of each of the three separate color channels for the second frame, and saves the rendering information of the second frame in a memory of the second display.
• a method of realizing video pipeline implementation of dynamically multiplexed foveated rendering with late time warp and raster scan output includes rendering a foveated image that includes a foveated zone and a peripheral zone, wherein the foveated zone has a first set of image data and is rendered at a first pixel resolution and the peripheral zone has a second set of image data and is rendered at a second pixel resolution lower than the first pixel resolution and generating a control packet that includes rendering information associated with the foveated image.
• the method further includes time warping the foveated image for movement in a viewer's position to form a time warped image, transmitting the time warped image and the control packet to a video processor, and performing a late time warping of the time warped image to form an updated image based on the latest viewer's pose data.
• the method also includes packing the updated image and the control packet to form a frame, transmitting the frame to a display unit, wherein the display unit parses the control packet from the frame, decodes the control packet to obtain the rendering information, and saves the rendering information of the frame in a memory of the display unit, and projecting a display image rendered according to the rendering information.
• FIG. 1 is an exemplary image illustrating an implementation of foveated rendering (FR).
• FIGS. 2A - 2E illustrate some example artifacts that can be caused by subsampling in FR.
• FIGS. 3A - 3C are spatial profiles illustrating a field of view and foveated rendering using temporal multiplexing according to some embodiments.
• FIGS. 3D - 3F are images illustrating native resolution, subsampling, and image blending according to an embodiment of the present invention.
• FIGS. 3G - 3H are text boxes illustrating subsampling and temporal multiplexing according to an embodiment of the present invention.
• FIG. 4 shows a simplified flowchart illustrating a method of generating foveated rendering using temporal multiplexing according to some embodiments.
• FIGS. 5A - 5D are spatial profiles illustrating a left field of view and a right field of view for foveated rendering using binocular multiplexing according to some embodiments.
• FIGS. 6A - 6C are spatial profiles illustrating a left field of view and a right field of view for foveated rendering using binocular multiplexing in combination with temporal multiplexing according to some embodiments.
• FIG. 7 shows a simplified flowchart illustrating a method of generating foveated rendering using binocular multiplexing according to some embodiments.
• FIG. 8 shows a simplified flowchart illustrating a method of generating foveated rendering using a combination of temporal multiplexing and binocular multiplexing according to some embodiments.
• FIG. 9 shows an exemplary control packet to be embedded in a video frame that provides FR information for use in a video pipeline according to some embodiments.
• FIG. 10 shows a block diagram of an exemplary video pipeline for dynamically multiplexed FR according to some embodiments.
• FIG. 11 shows a block diagram of an exemplary video pipeline for dynamically multiplexed FR that includes time warp according to some embodiments.
• FIG. 12 shows a block diagram of an exemplary video pipeline for dynamically multiplexed foveated rendering that includes time warp configured for a sequential color display according to some embodiments.
• FIG. 13 shows a block diagram of an exemplary video pipeline for dynamically multiplexed foveated rendering with late time warp and raster scan output according to some embodiments.
• a spatial profile for a field of view can be used to match the rendering quality to the visual acuity of human eyes.
• the spatial profile includes a foveated zone and one or more peripheral regions.
• the foveated zone would be rendered with maximum fidelity (e.g., at the native pixel resolution), while the peripheral regions would be rendered with lower resolutions.
• the spatial profile can be fixed within the FOV, or can be shifted based on eye gaze tracking data or based on content-estimations. Due to the inaccuracy and latency in eye gaze tracking or content-estimation, perceptual artifacts are often seen in FR.
• methods of FR using temporal multiplexing and/or binocular multiplexing of spatial profiles are provided. Such methods can improve the perceived visual quality of FR and reduce visual artifacts of FR.
• the methods can be used without eye gaze tracking, or can be used in conjunction with eye gaze tracking.
• Such methods can be implemented in a video pipeline from a graphics processor to a display unit using frame-specific control packets that provide information of FR for each frame. Thus, satisfactory visual quality of FR can be achieved while saving computation power and transmission bandwidth.
• FIG. 1 is an exemplary image illustrating an implementation of FR.
• the field of view (FOV) 110 is divided into two zones: a foveated zone 120 in the central region, and a peripheral zone 130 around the foveated zone 120.
• the foveated zone 120 is rendered with native resolution (illustrated with finer pixel grids), whereas the peripheral zone 130 is rendered with reduced resolution (illustrated with courser pixel grids).
• the location of the foveated zone 120 within the FOV 110 can be determined by measuring eye fixations using eye gaze trackers, or by inferring where the viewer is looking at on the basis of content. Alternatively, the location of the foveated zone 120 can be fixed, e.g., at the center of the FOV 110, assuming that the viewer is looking at the center of the FOV 110.
• the rendering pixel size in the peripheral zone 130 is often equal to an integer multiple of the pixel size in the foveated zone 120, e.g., through pixel binning or subsampling.
• a group of m * n native pixels can be replaced with one super pixel.
• a group of 2 * 2 native pixels are merged into one large pixel in the peripheral zone 130.
• the resolutions are halved in the peripheral zone 130 in both the horizontal and vertical dimensions compared to the resolutions in the foveated zone 120. This example is used in the discussions below.
• FIGS. 2A - 2E illustrate some example artifacts that can be caused by subsampling.
• a one-pixel-wide line 210 shown in FIG. 2A would become a two-pixel-wide line 220 shown in FIG. 2B. That is, the maximum spatial frequency is halved. Since adjacent pixels are combined, their luminance is averaged. Thus, the contrast is reduced, as illustrated in FIGS. 2A and 2B.
• Color may also be changed due to subsampling. For example, as illustrated in FIGS. 2C and 2D, a red line 230 and a green line 240 at the boundary between a red area and a green area may be combined into a wide yellow line 250.
• Subsampling may also reduce the legibility of fine text, as illustrated in FIG. 2E, in which the left side shows the text in native pixels and the right side shows the text in subsampled pixels.
• the above artifacts can be more noticeable when the content moves as a result of the viewer’s head motion or the motion of the content itself. For example, a boundary may be seen between the foveated zone and the peripheral zone, as indicated by brightness and/or contrast differences. When the content has a high contrast, for instance, white text on a black background seen with VR headsets or seen against a dark background with AR headsets, the above artifacts may be more noticeable.
• methods are provided to improve the perceptual quality of images rendered by FR by reducing the noticeability of the artifacts discussed above.
• a fixed spatial profile for rendering i.e., foveated native resolution vs. peripheral reduced resolution
• temporal and/or binocular multiplexing of varying spatial profiles are used for rendering. These methods are also referred to herein as temporal “dithering.”
• the solutions can have the advantage of being computationally light-weight, and yet not requiring accurate and fast eye gaze tracking.
• FIGS. 3A - 3C are spatial profiles illustrating a field of view and foveated rendering using temporal multiplexing according to some embodiments.
• FIG. 3 A shows a first spatial profile, in which a FOV 310 is divided into a first foveated zone 320 (represented by the dark pixels) and a first peripheral zone 330 (represented by the white pixels).
• FIG. 3B shows a second spatial profile, in which the FOV 310 is divided into a second foveated zone 340 and a second peripheral zone 350. As illustrated, the second foveated zone 340 is shifted with respect to the first foveated zone 320 in the horizontal direction (e.g., by 4 native pixels in the X direction).
• the first spatial profile and the second spatial profile are temporally multiplexed in a sequence of frames.
• the first spatial profile can be used for rendering odd frames
• the second spatial profile can be used for rendering even frames.
• the foveated zone is dynamically moved from frame to frame in a sequence of frames.
• images rendered at the native resolution are presented at all times.
• native-resolution images and subsampled images are presented alternatively from frame to frame.
• nativeresolution images and subsampled images may be blended into one as perceived by the viewer.
• the blending of the native-resolution images and the subsampled images can help to restore high-spatial frequencies and luminance contrast in the viewer’s visual perception.
• FIGS. 3D - 3F illustrate an example.
• FIGS. 3D - 3F are images illustrating native resolution, subsampling, and image blending according to an embodiment of the present invention.
• FIG. 3D shows a native resolution image that includes a one-pixel-wide line 360.
• FIG. 3E shows a subsampled image in which the one-pixel-wide line 360 becomes a two-pixel-wide line 370.
• FIG. 3F shows a result of blending the two images shown in FIGS. 3D and 3E, as may be perceived by a viewer. As illustrated in FIG. 3F, the high spatial resolution and contrast of the native resolution image are somewhat restored.
• FIGS. 3G - 3H are text boxes illustrating subsampling and temporal multiplexing according to an embodiment of the present invention.
• FIGS. 3G shows some text in a subsampled image.
• FIG. 3H shows the text in which a native resolution image and a subsampled image are multiplexed.
• the legibility of the text in FIG. 3H is improved as compared to FIG. 3G.
• the effective foveated zone e.g., the combined dark and grey area in FIG. 3C
• the location of the foveated zone can be spatially shifted between consecutive frames horizontally (e.g., in the X direction), or vertically (e.g., in the Y direction), or in both directions (e.g., combination of X and Y directions).
• the direction as well as the amount of the spatial shift can be varied dynamically.
• the frame rate may be limited by the capability of the display (e.g., a spatial light modulator or SLM).
• the frame rate can be 120 Hz or higher.
• the foveated zone can be spatially shifted to a set of predetermined locations in a fixed order or a random order, to cover as much of the FOV 310 as possible. Therefore, a viewer may perceive high quality images in the entire FOV, even when the viewer’s eye gaze changes.
• FIG. 4 shows a simplified flowchart illustrating a method 400 of generating foveated rendering using temporal multiplexing according to some embodiments.
• the method 400 includes, at 402, generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method 400 further includes, at 404, generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method 400 further includes, at 406, multiplexing the first spatial profile and the second spatial profile temporally in a sequence of frames, so that a viewer perceives images rendered in a region of the first foveated zone that does not overlap with the second foveated zone and/or in a region of the second foveated zone that does not overlap with the first foveated zone as rendered at the first pixel resolution.
• FIG. 4 provides a particular method of generating foveated rendering according to some embodiments. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 4 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added and some steps may be removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• FIGS. 5 A - 5D illustrate spatial profiles illustrating a left field of view and a right field of view for foveated rendering using binocular multiplexing according to some embodiments.
• the field of view (FOV) for the left eye 510 is divided into a first foveated zone 520 (represented by the grey pixels) and a first peripheral zone 530 (represented by the white pixels).
• the FOV for the right eye 540 is divided into a second foveated zone 550 and a second peripheral zone 560.
• the FOV for the left eye 510 and the FOV for the right eye 540 are identical. As illustrated, instead of being fixed in the center of the FOV, the first foveated zone 520 is shifted toward the right, and the second foveated zone 550 is shifted toward the left.
• FIG. 5B if the viewer’s eye fixation lands at the center of the FOV where the first foveated zone 520 and the second foveated zone 550 overlap, the viewer may see native-resolution images in both eyes in his or her central vision.
• FIG. 5A if the viewer’s eye fixation lands on the right side of the FOV, the viewer may see nativeresolution images in the left eye and subsampled images in the right eye in his or her central vision.
• FIG. 5C if the viewer’s eye fixation lands on the left side of the FOV, the viewer may see native-resolution images in the right eye and subsampled images in the left eye in his or her central vision.
• FIG. 5D illustrates the effective spatial profile.
• the viewer may see native-resolution images in both eyes.
• the viewer may see native-resolution images in only one of the eyes.
• the combined region (represented by the grey and dark pixels) at which native-resolution images are seen by at least one eye is larger than the foveated zone 520 or 550 in each individual spatial profile.
• the location of the foveated zone 520 and 550 can be shifted horizontally (e.g., in the X direction), or vertically (e.g., in the Y direction), or in both directions (e.g., combination of X and Y directions).
• FIGS. 6A - 6C are spatial profiles illustrating a left field of view and a right field of view for foveated rendering using binocular multiplexing in combination with temporal multiplexing according to some embodiments.
• a first spatial profile for the left FOV in a first frame, can have the foveated zone 610 shifted toward the right, and a second spatial profile for the right FOV can have the foveated zone 620 shifted toward the left.
• a third spatial profile for the left FOV can have the foveated zone 630 shifted toward the left
• a fourth spatial profile for the right FOV can have the foveated zone 640 shifted toward the right.
• the first frame as shown in FIG. 6A can be for every odd frame
• the second frame as shown in FIG. 6B can be for every even frame.
• a fifth spatial profile for the left FOV can have the foveated zone 650 shifted upward
• a sixth spatial profile for the right FOV can have the foveated zone 660 shifted downward.
• the direction as well as the amount of the spatial shift can be varied dynamically.
• the foveated zone for both the left FOV and the right FOV can be spatially shifted to a set of predetermined locations in a fixed order or a random order, to cover as much of the FOV as possible. Therefore, a viewer may perceive high quality images in the entire FOV, while saving bandwidth significantly.
• the movement of the foveated zones can be calculated using the minimal and maximum possible interpupillary distances (IPDs) to ensure that good visual results can be achieved for the targeted viewers.
• IPDs interpupillary distances
• the methods of temporal multiplexing and binocular multiplexing of spatial profiles can be applied to various types of FR implementations, including, e.g., fixed FR, FR with eye gaze tracking, or content-based FR.
• FR with eye gaze tracking the methods described herein can effectively extend the foveated region and hence reduce the artifacts produced by inaccurate eye gaze tracking.
• content-based FR the methods described herein can reduce the artifacts due to prediction errors.
• the methods described herein can help make a smooth transition in visual quality from the highest resolution in the foveated region, to a multiplexed resolution in the near-periphery, and to the subsampled resolution in the far-periphery.
• FIG. 7 shows a simplified flowchart illustrating a method 700 of generating foveated rendering using binocular multiplexing according to some embodiments.
• the method 700 includes, at 702, generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method 700 further includes, at 704, generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method 700 further includes, at 706, multiplexing the first spatial profile and the second spatial profile for a left eye and a right eye of a viewer, respectively, so that the viewer perceives images rendered in a region of the first foveated zone that does not overlap with the second foveated zone and/or in a region of the second foveated zone that does not overlap with the first foveated zone as rendered at the first pixel resolution.
• FIG. 7 provides a particular method of generating foveated rendering according to some embodiments. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 7 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added and some steps may be removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• FIG. 8 shows a simplified flowchart illustrating a method 800 of generating foveated rendering using a combination of temporal multiplexing and binocular multiplexing according to some embodiments.
• the method 800 includes, at 802, generating a first spatial profile for an FOV by dividing the FOV into a first foveated zone and a first peripheral zone.
• the first foveated zone will be rendered at a first pixel resolution
• the first peripheral zone will be rendered at a second pixel resolution lower than the first pixel resolution.
• the method 800 further includes, at 804, generating a second spatial profile for the FOV by dividing the FOV into a second foveated zone and a second peripheral zone.
• the second foveated zone is spatially offset from the first foveated zone.
• the second foveated zone will be rendered at the first pixel resolution, and the second peripheral zone will be rendered at the second pixel resolution.
• the method 800 further includes, at 806, generating a third spatial profile for the FOV by dividing the FOV into a third foveated zone and a third peripheral zone.
• the third foveated zone is spatially offset from the first foveated zone.
• the third foveated zone will be rendered at the first pixel resolution, and the third peripheral zone will be rendered at the second pixel resolution.
• the method 800 further includes, at 808, generating a fourth spatial profile for the FOV by dividing the FOV into a fourth foveated zone and a fourth peripheral zone.
• the fourth foveated zone is spatially offset from the third foveated zone.
• the fourth foveated zone will be rendered at the first pixel resolution, and the fourth peripheral zone will be rendered at the second pixel resolution.
• the method 800 further includes, at 810, multiplexing the first spatial profile and the second spatial profile for a left eye and a right eye of a viewer, respectively, in odd frames. [0063] The method 800 further includes, at 812, multiplexing the third spatial profile and the fourth spatial profile for a left eye and a right eye of a viewer, respectively, in even frames.
• FIG. 8 provides a particular method of generating foveated rendering according to some embodiments. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 8 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added and some steps may be removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.
• FR rendering described herein can be implemented in the hardware and/or software pipeline in various ways. For example, they can be implemented in the designs of AR/VR systems through firmware or middleware. Such implementations can be transparent to applications. Alternatively, they can be implemented in existing AR/VR systems through software in the operating system (OS) or individual applications.
• OS operating system
• a graphics driver e.g., a graphics processing unit or GPU
• a control packet can be embedded within the video frame that provides frame-specific information of the FR, including the pixel indexing of the foveated zone. The information provided by the control packet can assist a display unit (e.g., an SLM ASIC) to unpack the image data for each frame.
• a display unit e.g., an SLM ASIC
• FIG. 9 shows an exemplary control packet according to some embodiments.
• the control packet can be embedded in a video frame generated by a GPU.
• the control packet can include information such as whether the FR mode is enabled, the ratio of downsampling (e.g., 4: 1, 9: 1, 16:1, and the like), the indices of the start row and the start column of the foveated regions.
• the control packet can serve as a map for the display unit (e.g., by a SLM ASIC) to unpack the image data in the video frame.
• the ratio of downsampling can be dynamically changed from frame to frame. For example, the ratio can be 16: 1 for most frames, and be changed into 4:1 for those frames with text content.
• each video frame can include color channels for the three primary colors (e.g., red, green, and blue).
• Each color channel can independently have the FR mode enabled or disabled. For example, since the human eye has the strongest perception of green, it may be advantageous to have the green content in full resolution in the entire frame, and apply FR only to the red content and blue content.
• the foveated zone for each color channel can have its own ratio of downsampling, size, and locations with indices of start row and start column. For example, green content can have a lower ratio of downsampling than that of red and blue. Also, the foveated zones for the three color channels do not need to be aligned with respect to each other.
• control packet information that may be included in a control packet is not limited to the specific information discussed above. Any information or factors that may affect image rendering, processing, display and the like may be included in the control packet.
• FIG. 10 shows a block diagram of an exemplary video pipeline implementation of dynamically multiplexed foveated rendering according to some embodiments.
• An image 1020 e.g., a video frame
• the image 1020 includes a foveated zone 1022 rendered at a high resolution and a peripheral zone 1024 rendered at a low resolution.
• the image data (including the high resolution image data for the foveated zone 1022 and the low resolution image data for the peripheral zone 1024) are packed into a video frame 1030.
• the high resolution image data can be packed into a first image block 1034
• the low resolution image data can be packed into a second image block 1036.
• a control packet 1032 is then concatenated with the first image block 1034 and the second image block 1036.
• the control packet 1032 can include information about the FR rendering (e.g., as illustrated in FIG. 9).
• the video frame 1030 can be sent over to a display unit 1050 (e.g., an SLM ASIC) via a channel link 1040.
• the packing of the video frame with the control packet 1032 can significantly reduce the payload and the data rate required, thereby minimizing the bandwidth and power on the channel link 1040.
• the display unit 1050 can use a frame parser 1060 (e.g., a decoder) to parse the control packet 1032 from the first image block 1034 and the second image block 1036.
• the control packet 1032 can be in the first row (e.g., row zero) of the video frame 1030.
• a control packet decoder 1070 can decode the control packet 1032.
• the information provided by the control packet 1032 can then be used to map the image data in the first image block 1034 and the second image block 1036 to the foveated zone and the peripheral zone, respectively, in the video memory 1080.
• a large pixel can be mapped to several native pixels of the display (e.g., to four native pixels if the subsampling ratio is 4:1).
• the display unit 1050 performs this decoding process for each frame.
• the display unit 1050 projects the image saved in the video memory 1080 to a viewer (e.g., outputting photons via an SLM).
• time stamps can also be included in the video frame 1030 to help with synchronization and managing latency, as well as partial screen refresh tasks, blank modes, and the like.
• a GPU renders a foveated image and generates an associated control packet.
• a time warp function can be performed on the image data based on a latest pose prediction from a computer vision processor to provide an image to a display unit to reflect the viewer’s point of view based on the latest pose prediction.
• the time warp may be performed prior to sending the image data and the control packet to the display unit. As illustrated in FIG. 11, time warping may be performed at time warp block 1140 before the image data and the control packet is sent to video processor 1150 as well as at time warp and remap block 1152 of video processor 1150.
• the time warp may be performed by a computer vision processor coupled to the display unit.
• the foveated image is then sent to a wearable video processor, while the control packet is sent over a secondary data channel.
• the video processor then performs a late time warp (e.g., using the latest pose data from a sensor suite of the wearable device) and reformats the foveated image, with a foveated region and a low resolution region.
• This method can work well for a global refresh SLM that is capable of unpacking the image before display. For a color sequential display, red, green, and blue packed images can be generated and sent separately.
• FIG. 11 shows a block diagram of an exemplary video pipeline for dynamically multiplexed foveated rendering that includes time warp according to some embodiments.
• a GPU 1110 generates a foveated image 1120 and an associated control packet 1130 (e.g., as illustrated in FIG. 9).
• a time warp block 1140 performs a time warp function on the foveated image 1120 to account for movement in the viewer’s position.
• the image data and the control packet 1130 are then sent to a video processor 1150 of a wearable device via a headset link.
• a time warp and remap block 1152 can perform a late time warp using the latest pose data (e.g., from a sensor suite of the wearable device). For example, in cases in which there is significant head motion, the latest pose data can be used to update the boundaries.
• the time warp and remap block 1152 can also remap the foveated image 1120 into a foveated region data block 1154 and a low resolution region data block 1156.
• the control packet 1130 can be concatenated with the foveated region data block 1154 and the low resolution region data block 1156 to form a video frame, to be sent to the display unit 1160 (e.g., an SLM ASIC).
• the display unit 1160 can unpack the video frame using the control packet in a manner similar to that of the display unit 1050 illustrated in FIG. 10 and discussed above.
• FIG. 12 shows a block diagram of an exemplary video pipeline for dynamically multiplexed foveated rendering that includes time warp configured for a sequential color display according to some embodiments.
• the GPU 1210 renders a first foveated image 1212 for the left eye and a second foveated image 1214 for the right eye.
• An associated control packet 1216 is generated that provides information of the FR (e.g., as illustrated in FIG. 9) for both the first foveated image 1212 and the second foveated image 1214.
• a time warp block 1218 performs a time warp function on the first foveated image 1212 and the second foveated image 1214.
• a compression block 1219 performs compression of the image data.
• the compressed image data and the control packet 1216 are then sent to the video processor 1220 at the wearable device via a headset link.
• a decompression block 1222 decompresses the image data and recovers the first foveated image 1212 and the second foveated image 1214.
• a first time warp and remap block 1224 performs time warp on the first foveated image 1212 based on the latest pose data.
• the first time warp and remap block 1224 also maps the first foveated image 1212 into three separate color channels (e.g., red 1232a/1232b, green 1234a/1234b, and blue 1236a/1236b).
• the three color channels (i.e., 1232a, 1234a, and 1236a), along with the control packet, are packed as a first video frame 1228 to be sent to a first display 1230 (e.g., an LCOS display) for the left eye.
• a second time warp and remap block 1226 performs time warp on the second foveated image 1214 based on the latest pose data.
• the second time warp and remap block 1226 also maps the second foveated image 1214 into three separate color channels (i.e., 1232b, 1234b, and 1236b).
• the three color channels, along with the control packet are packed as a second video frame 1229 to be sent to a second display 1240 (e.g., an LCOS display) for the right eye.
• the first display 1230 can unpack the first video frame 1228 using the control packet in a manner similar to that of the display unit 1050 illustrated in FIG. 10 and discussed above.
• the image data for each of the three color channels 1232a, 1234a, and 1236a is saved in a video memory, to be projected to the viewer’s left eye.
• each of the three color channels 1232a, 1234a, and 1236a can have its independent foveated zone, downsampling ratio, and the like.
• the foveated zones for the three color channels 1232a, 1234a, and 1236a do not need to be aligned with respect to each other.
• the three color channels 1232a, 1234a, and 1236a can be displayed to the viewer sequentially.
• the second display 1240 can unpack the second video frame 1229 in a similar manner.
• FIG. 13 shows a block diagram of an exemplary video pipeline for dynamically multiplexed foveated rendering with late time warp and raster scan output according to some embodiments.
• a foveated image 1312 is rendered at the GPU 1310.
• a time warp block 1314 performs a time warp function to the foveated image 1312.
• a control packet 1316 that includes information about the FR rendering (e.g., as illustrated in FIG. 9) can be created for the foveated image 1312.
• the image data and the control packet 1316 are sent to a video processor 1320 of a wearable device via a headset link.
• a time warp and remap block 1324 can perform a late time warp to the foveated image using the latest pose data (e.g., from a sensor suite of the wearable device).
• the time warped image and the control packet 1316 are packed together as a video frame 1322, which is then sent to the display unit 1330.
• a frame parser 1334 can parse the control packet 1316 from the image data 1312.
• a control packet decoder 1336 can decode the control packet 1316 accompanying the image data 1312.
• the information provided by the control packet 1316 can then be used to map the image data 1312 to the video memory 1332. For example, a large pixel in the low resolution region can be mapped to several native pixels of the display (e.g., to four native pixels if the subsampling ratio is 4:1).
• the display unit 1050 can then project the image saved in the video memory 1332 to a viewer (e.g., outputting photons via an SLM).
• the display unit 1330 uses the FR information provided in the control packet 1316 for each frame, so as to ensure correct mapping.
• the image data 1312 is kept in the rasterized form throughout the pipeline, so that the image data can be scanned out as it is scanned in.
• the video memory 1332 can be a relatively small line buffer for feeding out the newly arrived image data; there is no need for a big buffer to keep the entire frame. Also, the latency can be kept relatively low.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Human Computer Interaction (AREA)
• Optics & Photonics (AREA)
• Geometry (AREA)
• Computer Graphics (AREA)
• Ophthalmology & Optometry (AREA)
• Animal Behavior & Ethology (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Medical Informatics (AREA)
• Molecular Biology (AREA)
• Surgery (AREA)
• Biophysics (AREA)
• General Health & Medical Sciences (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Computer Hardware Design (AREA)
• Processing Or Creating Images (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Image Processing (AREA)
• Controls And Circuits For Display Device (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=85200358

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (3)

Citations (5)

Family Cites Families (2)

• 2022
  • 2022-08-12 JP JP2024508432A patent/JP2024531196A/ja active Pending
  • 2022-08-12 WO PCT/US2022/040267 patent/WO2023018997A1/en not_active Ceased
  • 2022-08-12 CN CN202280055748.4A patent/CN117859320A/zh active Pending
  • 2022-08-12 EP EP22856697.2A patent/EP4385202A4/en active Pending
• 2024
  • 2024-02-12 US US18/439,296 patent/US12347062B2/en active Active
• 2025
  • 2025-05-23 US US19/217,669 patent/US20250285213A1/en active Pending

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events