US20200018978A1 - Dynamic Panel Masking - Google Patents
Dynamic Panel Masking Download PDFInfo
- Publication number
- US20200018978A1 US20200018978A1 US16/033,162 US201816033162A US2020018978A1 US 20200018978 A1 US20200018978 A1 US 20200018978A1 US 201816033162 A US201816033162 A US 201816033162A US 2020018978 A1 US2020018978 A1 US 2020018978A1
- Authority
- US
- United States
- Prior art keywords
- size value
- panel
- hmd
- rendered
- mask
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0189—Sight systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/18—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for optical projection, e.g. combination of mirror and condenser and objective
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
- G06F3/0312—Detection arrangements using opto-electronic means for tracking the rotation of a spherical or circular member, e.g. optical rotary encoders used in mice or trackballs using a tracking ball or in mouse scroll wheels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
- G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2320/00—Control of display operating conditions
- G09G2320/02—Improving the quality of display appearance
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
Definitions
- VR virtual reality
- Displays for VR systems such as those embedded in a VR headset, typically operate at a minimum refresh rate that is suitable for VR applications. For instance, 90 Hertz (Hz) is a common refresh rate for VR displays.
- Hz 90 Hertz
- a graphics-based application such as a video game, outputs frames for rendering at a frame rate that matches the refresh rate of the display, meaning that a new frame received from the application (referred to herein as an “actual frame”) is displayed at every screen refresh.
- Such a live rendering scenario is often referred to as the application “hitting frame rate.”
- an application does not always hit frame rate for various reasons.
- the application may intermittently drop a frame, and/or the application may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second).
- a technique called “rotation-only re-projection” can be used to replace missing frames with re-projected frames in a way that accounts for the user's head rotation, making it appear to the user as if the application is hitting frame rate.
- a deficient frame rate from the application may cause in-game stuttering or hitching.
- re-projection is a technique that allows for a better user experience when an application is not hitting frame rate.
- the application is outputting frames at half the ideal frame rate (e.g., 45 frames per second where 90 frames per second is the ideal frame rate).
- every other frame can be re-projected using pixel data from the most recently-rendered actual frame to create a re-projected frame that transforms the scene (e.g., through rotation and re-projection calculations) to match the re-projected scene to the user's current head orientation.
- This makes it look to the user as if the scene is moving in a way that is expected given the user's head rotation, even when re-projected frames are used to compensate for the application failing to hit frame rate.
- rotation-only re-projection prevents in-game stuttering or hitching, it produces its own unwanted visual artifacts during head rotation, at least in VR systems that use low-persistence displays (e.g., where the display is illuminated for a small fraction of the frame time).
- rotation-only re-projection can cause a strobing flash to appear at the leading edge of each display panel in the direction of head rotation when the user is rotating his/her head. This strobing flash occurs during head rotation because the pixels at the leading edge of the display panel toggle every frame between an actual frame's valid pixels and solid black pixels in the following re-projected frame.
- the black pixels occur during head rotation in the re-projected frames because there is no pixel data available from the previously-rendered actual frame to use for the pixels at the leading edge of the display panel for the re-projected frame. Accordingly, when the user rotates his/her head while re-projection is being used, the leading edges of the display panels in the direction of the user's head rotation toggles every frame between black pixels and illuminated pixels, causing an unwanted strobing artifact at the leading edges of the display panels.
- FIG. 1A is a diagram illustrating example left and right display panels of a head-mounted display (HMD) where a frame is rendered with a dynamically expanded panel mask during head rotation while frames are being processed using re-projection, the direction of head rotation being in a rightward (or positive X) direction.
- HMD head-mounted display
- FIG. 1B is a diagram illustrating the example left and right display panels of the HMD of FIG. 1A where a frame is rendered with a dynamically expanded panel mask during head rotation while frames are being processed using re-projection, the direction of head rotation being in a leftward (or negative X) direction.
- FIG. 2 is a diagram illustrating example left and right display panels of a HMD showing a difference between a fully contracted state of the panel masks and a fully expanded state of the panel masks.
- FIG. 3 is another diagram illustrating example left and right display panels of a HMD showing the difference between a fully contracted state of the panel masks and a fully expanded state of the panel masks.
- FIG. 4 is a flow diagram of an example process for dynamically re-sizing a portion of a panel mask on a display panel of a HMD, in accordance with embodiments disclosed herein.
- FIG. 5 is a flow diagram of a more detailed example process for dynamically re-sizing a portion of a panel mask on a display panel of a HMD, in accordance with embodiments disclosed herein.
- FIG. 6 illustrates example components of a wearable device, such as a VR headset, in which the techniques disclosed herein can be implemented.
- a HMD may be worn by a user for purposes of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment.
- VR virtual reality
- AR augmented reality
- One or more display panels of the HMD render images based on frames that are output by an application (e.g., a video game), and these images are viewed by a user through the optics that are included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment.
- a panel mask(s) may be rendered at a periphery of the display panel(s) of the HMD to provide a border around the images presented on the display panel(s).
- the panel mask frames the image much like a picture frame.
- the inner edges of the panel mask(s) can be blurred (e.g., using a subtle vignette) to make the panel mask appear to the user as a near-field object that is close to (or disposed on) on the user's face, much like facial features (e.g., the nose bridge between the eyes) are seen when focusing on far-field objects.
- the HMD may utilize a technique called “re-projection” to compensate for the application not hitting frame rate.
- re-projection a technique called “re-projection” to compensate for the application not hitting frame rate.
- a user wearing the HMD may rotate his/her head, which, as mentioned, may cause an unwanted strobing artifact to appear at the leading edge of the display panel(s) that is presently not covered by the panel mask.
- the HMD can detect when re-projection is, or is not, being used, and at least a portion of the panel mask can be dynamically adjusted in size based upon whether re-projection is, or is not, being used.
- a portion of the panel mask may be adjusted (e.g., contracted away from a center of the display panel or expanded towards the center of the display panel) by an appropriate amount depending on an amount of rotation of the HMD during re-projection. This allows for covering up (or hiding) unwanted visual artifacts, such as a strobing flash, that may manifest during re-projection coupled with head movement.
- the portion of the panel mask at the leading (e.g., right) edge of the display panel(s) may be increased in size to ensure that the unwanted strobing artifact is covered by the panel mask so that the user does not see the strobing flash in the image area of the display panel.
- this expanded portion of the panel mask can be adjusted in size over time back to a fully contracted state so that more of the scene is rendered in the image area of the display panel.
- This technique of dynamically expanding and contracting at least a portion of the panel mask, based on head rotation coupled with re-projection, allows for a better viewing experience for a user of the HMD because the user does not notice the unwanted strobing artifact caused by re-projection during head rotation; the panel mask expands to occlude the strobing artifact when it is present, and the panel mask contracts after the strobing artifact has ceased to reveal more of the scene in the image area of the display panel.
- An example process for dynamically adjusting a size of a portion of a panel mask includes determining, by one or more processors, whether a series of frames are being rendered on a display panel of a HMD using re-projection, determining, based at least in part on whether the series of frames are being rendered using re-projection, a target size value for at least a portion of a panel mask, determining that a size value associated with at least the portion of the panel mask is set to an existing size value that is not equal to the target size value, adjusting the size value from the existing size value to the target size value to increase or decrease a size of at least the portion of the panel mask from a current size to an adjusted size, and rendering a frame, of the series of frames, on the display panel along with the panel mask rendered at a periphery of the display panel, wherein at least the portion of the panel mask is rendered at the adjusted size in accordance with the target size value.
- the size of the portion of the panel mask can be subsequently decreased in size if the use of re-projection ceases and/or if head rotation slows or ceases altogether. For example, if the use of re-projection ceases, the portion of the panel mask can be decreased in size incrementally over a time period to a minimum size or some other target value. In this way, the contraction of the panel mask back to its fully contracted state is not visually distracting to the user.
- HMD configured to implement the techniques and processes disclosed herein, as well as non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein.
- non-VR applications e.g., AR applications
- non-gaming applications such as industrial machine applications, defense applications, robotics applications, and the like.
- FIG. 1A is a diagram illustrating an example head-mounted display (HMD) 100 that includes a left display panel 102 (L) and a right display panel 102 (R), which make up a pair of display panels.
- HMD head-mounted display
- the example HMD 100 includes two display panels, 102 (L) and 102 (R)
- the HMD 100 may include a single display panel 102 , or more than two display panels 102 .
- “display panel 102 ,” as used herein, may refer to either of the display panels 102 (L) or 102 (R) of a two-panel HMD 100 , as shown in FIG. 1A , or it may refer to a single display panel 102 of a HMD 100 with any number of display panels (e.g., a single-panel HMD 100 or a multi-panel HMD 100 ).
- the display panel(s) 102 may represent any suitable type of display, such as an emissive display that utilizes light emitting elements to emit light during presentation of image frames (herein referred to as “frames”) on the display panel(s) 102 .
- the left and right display panels 102 (L) and 102 (R) may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display that utilizes a suitable display technology for HMD applications.
- LCDs liquid crystal displays
- OLED organic light emitting diode
- ILED inorganic light emitting diode
- the HMD 100 may represent a VR headset for use in VR systems, such as for use with a VR gaming system. However, the HMD 100 may additionally, or alternatively, be implemented as an AR headset for use in AR applications.
- AR a user sees virtual objects overlaid on a real-world environment
- VR the user does not see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panels 102 and the optics (e.g., lenses) of the HMD 100 . Examples described herein pertain primarily to a VR-based HMD 100 , but it is to be appreciated that the HMD 100 is not limited to implementation in VR applications.
- an application executing on a computing device such as the HMD 100 itself, or a computing device (e.g., a personal computer (PC), game console, etc.) associated with, and coupled to, the HMD 100 —may be configured to output a series of frames 106 that are ultimately rendered on the display panel(s) 102 of the HMD 100 .
- the image data of the frames 106 is presented within an image area 108 of the display panel(s) 102 that is not covered by a panel mask 110 .
- FIG. 1A shows a left image area 108 (L) at the center of the left display panel 102 (L) and inside a left panel mask 110 (L), and a right image area 108 (R) at the center of the right display panel 102 (R) and inside a right panel mask 110 (R).
- the series of frames 106 may be output from a video game application, or any other type of graphics-based application.
- the application may be executed in a graphics pipeline that outputs per-pixel values (e.g., color values) that are combined to create on image (within the image area 108 ) on the display panel(s) 102 with a desired visual effect.
- per-pixel values e.g., color values
- a head tracking module of the HMD 100 may generate data about the position and pose of the HMD 100 that is provided to the application in order to inform the application regarding how to render a next frame 106 in the series of frames 106 in accordance with the user's 104 head position/pose.
- a left panel mask 110 (L) is rendered at a periphery of the left display panel 102 (L) and a right panel mask 110 (R) is rendered at a periphery of the right display panel 102 (R).
- panel mask 110 may refer to either of the panel masks 110 (L) or 110 (R), as shown in FIG. 1A .
- a single-panel HMD 100 may include a single panel mask 110 that surrounds the image area 108 of a single display panel 102 .
- the HMD 100 may compute a mesh for the panel mask(s) 110 , which is used to render the panel mask(s) 110 in a fully contracted state at the periphery of the display panel(s) 102 , meaning that the portions of the panel mask(s) 110 that are dynamically adjustable in size are initially rendered at a smallest size to reveal more of the scene within the image area(s) 108 .
- the panel mask(s) 110 may be persistently rendered along with each frame, although the size of individual portions of the panel mask(s) 110 may be adjusted in particular ones of the frames to hide unwanted visual artifacts, as needed.
- the panel mask(s) 110 may be implemented as a plurality of black pixels at the periphery of the display panel(s) 102 , which may be added to the frame buffer before rendering a frame.
- the panel mask(s) 110 is depicted in the figures as a generally annular-shaped panel mask that encloses a generally circular-shaped image area 108 , it is to be appreciated that other geometries can be used to implement the panel mask(s) 110 , such as by rendering the panel mask(s) 110 as a square, rectangular, or other geometrically-shaped frame surrounding the image area 108 .
- the panel mask(s) 110 may be implemented as three separate panel masks corresponding to red, green, and blue color channels, wherein each separate panel mask 110 of the three panel masks 110 is shaped based on a known chromatic distortion of the optics included in the HMD 100 to mitigate effects of color fringing, which may occur at or near the outer edge of the display panel 102 .
- a known chromatic distortion of the optics included in the HMD 100 to mitigate effects of color fringing, which may occur at or near the outer edge of the display panel 102 .
- chromatic distortion e.g., red, green, blue (RGB) separation
- the user 102 may see cyan fringing on the panel edges, which comes from the chromatic distortion that separates RGB, where the red pixels are chromatically distorted the most (toward the center of the display panel 102 ), followed by a lesser amount of distortion for the green pixels, and the least amount of distortion for the blue pixels (e.g., the blue is distorted the least, remaining closest to the edge of the panel, causing the cyan fringing effect).
- each display panel 102 may independently mask red, green, and blue to mitigate the color fringing effect.
- FIG. 1A shows the left panel mask 110 (L) as a relatively dark grey area, and the right panel mask 110 (R) as a relatively light grey area.
- the pair of panel masks 110 (L) and 110 (R) may be rendered in a uniform color (e.g., black pixels) on both display panels 102 (L) and 102 (R).
- the user 104 perceives the two panel masks 110 (L) and 110 (R) as a single panel mask that surrounds the two image areas 108 (L) and 108 (R), much like a human perceives an environment with eye sockets and a nose bridge that surround each eye in a near field of view.
- each panel mask 110 (L) and 110 (R) may be blurred slightly (e.g., using a subtle vignette) to mimic near-field facial features, which is more familiar to humans than a sharp inner edge 112 would be.
- each of the panel masks 110 (L) and 110 (R) are dynamically adjustable in size to expand and contract portions of the panel mask 110 to cover more or less of the image area 108 , as needed, for each frame 106 of the series of frames 106 .
- a portion of the panel mask(s) 110 may increase in size by expanding inward toward a center of the display panel(s) 102 , which allows for covering up (or hiding) unwanted visual artifacts that manifest at that portion of the display panel(s) 102 during re-projection coupled with head rotation.
- the display panel(s) 102 of the HMD 100 may operate at a refresh rate of say, 90 Hz, which is a common refresh rate for VR displays.
- 90 Hz is used as an example refresh rate
- the display panel(s) 102 may operate at other, different refresh rates, without changing the basic characteristics of the techniques and systems disclosed herein.
- the application that outputs the series of frames 106 may output actual frames 106 (A) at a frame rate that matches the refresh rate of the display panel(s) 102 (e.g., actual frames 106 (A) are output at 90 frames per second).
- the HMD 100 may be configured to employ re-projection (e.g., rotation-only re-projection) to replace missing actual frames 106 (A) with re-projected frames 106 (R), and the HMD 100 may do so in a way that accounts for the user's 104 head rotation, making it appear to the user 104 as if the application is hitting frame rate and that the scene is moving about the display panels 102 in an expected manner given the user's 104 head rotation.
- re-projection e.g., rotation-only re-projection
- FIG. 1A illustrates this re-projection scenario—where the series of frames 106 includes an actual frame 106 (A)( 1 ), followed by a re-projected frame 106 (R)( 2 ), followed by another actual frame 106 (A)( 3 ), and so on and so forth.
- the application may be outputting frames at half of the ideal frame rate (e.g., at a frame rate of 45 frames per second, where 90 frames per second is the ideal frame rate).
- every other frame between sequential actual frames 106 (A) is a re-projected frame 106 (R).
- a re-projected frame 106 (R) is generated using pixel data from a recently-rendered actual frame 106 (A) (e.g., the most recently-rendered actual frame 106 (A)) to create a scene that is transformed (e.g., through rotation and re-projection calculations) in a way that accounts for the user's 104 head rotation. Assuming that the actual frame 106 (A)( 1 ) is rendered before the re-projected frame 106 (R)( 2 ) in FIG. 1A , the re-projected frame 106 (R)( 2 ) can be derived from the pixel data associated with the previously-rendered actual frame 106 (A)( 1 ).
- the scene of the previously-rendered actual frame 106 (A)( 1 ) is rotated and re-projected to create the re-projected frame 106 (R)( 2 ) in a manner that the user 104 would expect given his/her head movement.
- a strobing flash may appear at the leading edge of each display panel 102 in the direction of head rotation.
- the user 104 is shown as rotating his/her head in a rightward (i.e., positive X) direction of rotation.
- the aforementioned strobing flash may occur during re-projection, and it may manifest at the right edge (i.e., the leading edge in the direction of rotation) of each display panel 102 (L) and 102 (R) because there is no pixel data available in the previously-rendered actual frame 106 (A)( 1 ) to render the pixels at the right edge of the each display panel 102 (L) and 102 (R) for the re-projected frame 106 (R)( 2 ) since the user 106 has rotated in the rightward direction.
- the HMD 100 may be configured to dynamically increase the size of a right portion of each panel mask 110 to cover up (or hide) this unwanted visual artifact. Without changing the size of the right portion of each panel mask 110 , this strobing artifact may be visible within the image area(s) 108 of the display panel(s) 102 .
- the HMD 100 may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to determine whether a series of frames 106 are being rendered on a display panel(s) 102 of the HMD 100 using re-projection. That is, the logic may determine whether one or more of the frames 106 in the series of frames 106 are re-projected frames 106 (R) generated from pixel data associated with actual frames 106 (A) received from an application that have already been rendered on the display panel(s) 102 . In some embodiments, the logic may make this determination upon rendering the re-projected frame 106 (R)( 2 ).
- logic e.g., software, hardware, and/or firmware, etc.
- the logic may make this determination after rendering the actual frame 106 (A)( 1 ), but before rendering the re-projected frame 106 (R)( 2 ).
- the logic may also be configured to determine, based at least in part on rotation data provided by a head tracking module of the HMD 100 indicating that the HMD 100 has rotated (e.g., between time, t 1 , and time, t 2 ), a target size value for each adjustable portion of the panel mask(s) 110 .
- the logic may respond by adjusting a size value associated with at least a portion of the panel mask(s) 110 from the existing size value to the target size value to increase a size of at least the portion of the panel mask 110 from a current size to an adjusted size, and by rendering a frame (e.g., the actual frame 106 (A)( 3 )), of the series of frames 106 , on the display panel(s) 102 along with the panel mask(s) 110 rendered at a periphery of the display panel(s) 102 , wherein at least the portion of the panel mask(s) 110 is rendered at the adjusted size.
- a frame e.g., the actual frame 106 (A)( 3 )
- FIG. 1A shows that a right portion of the left panel mask 110 (L) and a right portion of the right panel mask 110 (R)—each portion corresponding to the rightward direction of rotation—increase in size from a current size to an increased size. This is indicated in FIG.
- FIG. 1B is a diagram illustrating the example left and right display panels 102 of the HMD 100 of FIG. 1A , where a frame 106 (e.g., the actual frame 106 (A)( 3 )) is rendered with a dynamically expanded panel mask(s) 110 during head rotation while frames 106 are being processed using re-projection, the direction of head rotation being in a leftward (or negative X) direction in FIG. 1B , which is opposite to the direction of rotation shown in FIG. 1A .
- the logic of the HMD 100 is configured to respond similarly to how it responds in the scenario of FIG.
- each panel mask 110 is resized in response to the different direction of rotation that is detected by the HMD 100 .
- the left portion of each panel mask 110 is pulled inward to expand the size of the left portion of the panel mask 110 so that more of the image area 108 on the left edge/side of the display panel(s) 102 is covered by the panel mask(s) 110 .
- each panel mask 110 can dynamically increase in size (e.g., by expanding inward toward the center of the display panel(s) 102 ) so that more of the image area 108 on the top/bottom edge/side of the display panel(s) 102 is covered by the panel mask(s) 110 .
- a rotational directionality of the HMD's 100 is a vector having both X and Y components, this may cause adjacent portions of the panel mask 110 to simultaneously expand in size. For example, if the user 104 were to rotate his/her head diagonally into the positive X-Y quadrant of the reference coordinate system shown in FIGS.
- both the right portion and the top portion of the panel mask(s) 110 may be simultaneously expanded in size such that the two adjacent portions of the panel mask 110 expand inward toward the center of the display panel 102 at the same time. It can also be appreciated that, in some implementations, the entire panel mask(s) 110 (i.e., all portions) may be expanded in size, causing the image area(s) 108 to contract (or get smaller) as more of the image area(s) 108 is covered by an expanded panel mask(s) 110 .
- the image area(s) 108 may dilate back to a maximum size of the image area(s) 108 as less of the image area(s) 108 is covered by a contracted panel mask(s) 110 .
- FIG. 2 is a diagram illustrating example left and right display panels 102 of a HMD 100 showing a difference between a fully contracted state of a panel mask(s) 110 at the top of FIG. 2 and a fully expanded state of the panel mask(s) 110 at the bottom of FIG. 2 .
- individual portions of the panel mask 110 for each display panel 102 can be dynamically adjusted in size, as needed, to cover more or less of the image area 108 of each display panel 102 . There may be limits or bounds for these size adjustments. For example, an individual portion of the panel mask 110 may be increased up to, but not beyond, a maximum size, and decreased down to, but not below, a minimum size.
- the panel mask 110 can be persistently rendered with each frame, and its size is controllable based on values that range between the minimum size value (e.g., “0”) and the maximum size value (e.g., “1”).
- the size value for an individual portion of the panel mask 110 is clamped between a minimum size value (e.g., “0”) and a maximum size value (e.g., “1”), but is adjustable to any number between these two bounds, inclusive of these two bounds.
- a percentage adjustment e.g., 0 is 0%, 0.4 is 40%, 0.6 is 60%, 1 is 100%, etc.
- this size adjustment can be controlled independently for individual portions of the panel mask 110 .
- the left panel mask 110 (L) For instance, consider partitioning the left panel mask 110 (L) into four portions including a left portion 110 (L)( 1 ), a top portion 110 (L)( 2 ), a right portion 110 (L)( 3 ), and a bottom portion 110 (L)( 4 ).
- the right panel mask 110 (R) may be partitioned into four portions including a left portion 110 (R)( 1 ), a top portion 110 (R)( 2 ), a right portion 110 (R)( 3 ), and a bottom portion 110 (R)( 4 ).
- Each of these portions of the panel mask 110 can be adjusted in size between a minimum size value (e.g., a size value set to “0”) corresponding to the fully contracted state at the top of FIG.
- each panel mask 110 there may be at least four dynamically adjustable size values for each panel mask 110 to control the size of the individual portions of the panel mask 110 .
- the inner edge 112 of the panel mask 110 (at each portion of the panel mask 110 ) can be moved partway between the minimum extent (shown at the top of FIG. 2 ) and the maximum extent (shown at the bottom of FIG. 2 ) by setting the size value to a value between the minimum size value and the maximum size value (e.g., a size value between “0” and “1”).
- the panel mask(s) 110 may be persistently rendered at a minimum size (i.e. in the fully contracted state) regardless of whether re-projection is being used and/or head rotation is present.
- some HMDs 100 may exhibit color fringing at or near the outer edge of the display panel 102 , and it may be desirable to cover up (or hide) such visual artifacts so that they are not a distraction to the user 104 .
- the panel mask(s) 110 need to expand in size beyond the maximum size value because there are practical limits to the rate of rotation of the HMD 100 when it is worn by a user 104 , and there are practical limits to the perceivable visual elements when the user 104 is rotating his/her head at high speeds.
- the panel mask 110 does not need to be expanded to cover the entirety of the display panel 102 in this regard; the expansion of the panel mask can “bring in” the field of view just enough to cover unwanted visual artifacts.
- the user 104 would not be able to discern the unwanted visual artifacts that manifest themselves at those high speeds of rotation because the eye simply can't track these visual elements when the field of view is changing at a rapid rate.
- FIG. 3 is another diagram illustrating example left and right display panels 102 (L) and 102 (R) of a HMD 100 showing the difference between a fully contracted state of the panel masks 110 (L) and 110 (R) and a fully expanded state of the panel masks 110 (L) and 110 (R).
- the fully contracted state is shown by the dashed lines that represent where the inner edges 112 of the panel masks 110 would be in the fully contracted state.
- the individual portions of the panel mask 110 may have their size values set to zero at startup of the HMD 100 .
- the fully expanded state of the panel mask(s) 110 is shown by the solid lines that represent where the inner edges 112 of the panel masks 110 would be in the fully expanded state.
- the individual portions of the panel mask 110 can be adjusted to an intermediate size between the minimum size and the maximum size by setting the size value for each portion to an intermediate value between the minimum size value (e.g., “0”) and the maximum size value (e.g., “1”).
- the processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic).
- the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations.
- computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
- the order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.
- FIG. 4 is a flow diagram of an example process 400 for dynamically re-sizing a portion of a panel mask 110 on a display panel 102 of a HMD 100 , in accordance with embodiments disclosed herein.
- the process 400 is described with reference to the previous figures.
- logic of a HMD 100 may determine a target size value(s) for at least a portion(s) of a panel mask(s) 110 .
- the logic of the HMD 100 may, for example, determine four target size values corresponding to a left portion 110 (L)( 1 ), a top portion 110 (L)( 2 ), a right portion 110 (L)( 3 ), and a bottom portion 110 (L)( 4 ) of a left panel mask 110 (L), as well as four target size values corresponding to a left portion 110 (R)( 1 ), a top portion 110 (R)( 2 ), a right portion 110 (R)( 3 ), and a bottom portion 110 (R)( 4 ) of a right panel mask 110 (R).
- a total of eight target size values may be determined at block 402 for a two-panel HMD 100 that renders a left panel mask 110 (L) and a right panel mask 110 for a given frame 106 .
- the determination of the target size value(s) at block 402 may depend on whether re-projection is, or is not, currently being used to render the series of frames 106 . If re-projection is ongoing (e.g., if the given frame 106 is part of a series of frames 106 that are being rendered using re-projection), then the target size value(s) may be determined based at least in part on rotation data provided by a head tracking module of the HMD 100 .
- the target size value(s) may be set to a value(s) of zero at block 402 .
- the target size value for an individual portion of the panel mask 110 may be clamped between a minimum size value (e.g., “0”) and a maximum size value (e.g., “1”) such that the target size value can be set to any value between these two bounds (e.g., between “0” and “1”), inclusive of these two bounds.
- the target size value may be calculated as a value that corresponds to an amount that the HMD 100 has rotated. For instance, greater amounts of HMD 100 rotation correspond to target size values that are closer to the maximum size value (e.g., “1”), and lesser amounts of HMD 100 rotation correspond to target size values closer to the minimum size value (e.g., “0”).
- the logic of the HMD 100 may adjust a size value associated with at least the portion of the panel mask 110 from an existing size value to the target size value determined at block 402 , assuming that the target size value is not equal to the existing size value. That is, if the target size value is equal to the existing size value, then the logic may refrain from making a size adjustment at block 404 . Accordingly, if re-projection is ongoing, and if the logic determines that the size value associated with the portion of the panel mask is set to an existing size value that is less than the target size value, the adjustment at block 404 may result in a size of at least the portion of the panel mask 110 increasing from a current size to an increased size.
- the adjustment at block 404 may result in a size of at least the portion of the panel mask 110 decreasing from a current size to a decreased size.
- a downward size adjustment to a decreased size at block 404 may be a slow interpolation from a current size value to a lesser target value (e.g., the minimum value or some other target value) by a small amount (e.g., a per-frame lerp value of 0.005).
- a downward size adjustment at block 404 is further conditioned on having not increased the size of the portion of the panel mask 110 over the course of a predetermined number of consecutive frames 106 preceding the current frame 106 . For example, if the size of the portion of the panel mask 110 has not increased in the last ten consecutive frames, the size may be adjusted downward to a decreased size. Otherwise, if this consecutive frame criterion is not met, there may not be any adjustment to the size value at block 404 .
- the logic of the HMD 100 may render a frame 106 , of the series of frames 106 , on the display panel 102 along with the panel mask 110 rendered at a periphery of the display panel 102 , wherein at least the portion of the panel mask 110 is rendered at a size (e.g., an adjusted size) in accordance with the target size value.
- a size e.g., an adjusted size
- the process 400 allows for hiding unwanted visual artifacts that may occur during re-projection coupled with head rotation by virtue of the panel mask 110 expanding to cover the area of the display panel 102 where the unwanted visual artifacts would otherwise present themselves. This is shown, by way of example and not limitation, in FIGS. 1A and 1B , where a portion of the panel mask 110 on each display panel 102 is increased in size from a current size to an increased size and rendered with a frame 106 on the display panel(s) 102 .
- FIG. 5 is a flow diagram of a more detailed example process 500 for dynamically re-sizing a portion of a panel mask 110 on a display panel 102 of a HMD 100 , in accordance with embodiments disclosed herein.
- the process 500 is described with reference to the previous figures.
- a frame 106 of a series of frames 106 , is processed for rendering on a display panel(s) 102 of a HMD 100 .
- the panel mask(s) 110 that is to be rendered on the display panel(s) 102 along with the frame 106 may have size values associated with individual portions of the panel mask(s) 110 set to existing size values that dictate whether those portions of the panel mask(s) 110 are in a fully contracted state, a fully expanded state, or somewhere in between.
- the existing size values for each portion may be set to a minimum size value (e.g., a size value of zero) corresponding to a fully contracted state.
- logic of the HMD 100 may determine whether a most recently-rendered frame 106 , of the series of frames 106 , is a re-projected frame 106 (R) that was derived from pixel data associated with a previously-rendered actual frame 106 (A) that was received from an application. Said another way, the logic of the HMD 100 determines whether re-projection is occurring at present. If, at 504 , it is determined that the series of frames 106 are being rendered on the display panel(s) 102 of the HMD 100 using re-projection, the process 500 may follow the “YES” route from block 504 to block 506 .
- logic of the HMD 100 may determine, based at least in part on rotation data provided by a head tracking module of the HMD 100 , target size values for individual portions of the panel mask 110 . For example, four target size values may be determined, based on HMD 100 rotation, clamped 0.0 to 1.0 per direction.
- a first target size value between 0 and 1 (inclusive) may be determined for a left portion of the panel mask 110
- a second target size value between 0 and 1 (inclusive) may be determined for a top portion of the panel mask 110
- a third target size value between 0 and 1 (inclusive) may be determined for a right portion of the panel mask 110
- a fourth target size value between 0 and 1 (inclusive) may be determined for a bottom portion of the panel mask 110 .
- This may be done for each panel mask 110 , which means that a total of eight target size values may be determined at block 506 for left and right panel masks 110 (L) and 110 (R).
- the direction and the amount of HMD 100 rotation dictates the portion of the panel mask 110 and the target size value, respectively, for that portion of the panel mask 110 .
- the logic of the HMD 100 may determine, based on the rotation data provided by the head tracking module of the HMD 100 , an amount of rotation of the HMD 100 per direction (e.g., amounts of rotation in left, right, up, and down directions) over a period of time since a rendered frame 106 , of the series of frames 106 , was last rendered. Any suitable manner of determining an amount of rotation in any unit of measurement may be utilized for the determination at block 508 . For instance, a number of degrees of rotation between frames can be determined as the amount of rotation. In some embodiments, an amount of rotation determined at block 508 may be in the form of angular velocity.
- Directionality of the HMD's 100 rotation may be determined with reference to positive or negative horizontal and vertical directions, or any other suitable form of directional measurement, and the directionality of the HMD's 100 rotation may be broken down into component directions (e.g., left, right, up, and/or down directions in the X-Y plane). Accordingly, the logic at block 506 may include determining a portion(s), among multiple portions, of the panel mask(s) 110 that corresponds to the direction in which the HMD 100 has rotated. In general, the greater the amount of rotation of the HMD 100 , the greater the target size value so that more of the image area 108 on the display panel 102 is covered by the panel mask 110 for a greater amount of rotation.
- the target size value there may be a threshold amount of rotation that corresponds to a maximum size value for the portion of the panel mask 110 , and if the amount of rotation of the HMD 100 meets or exceeds this threshold amount, the maximum size value may be selected as the target size value at block 506 . If the amount of rotation of the HMD 100 is less than this threshold amount, the target size value is determined as something less than the maximum size value. In some embodiments, the target size value may be determined based on the relative differences of the re-projection transforms used for the previous frame 106 and the current frame 106 .
- a relative difference between the re-projection transforms used during the previous frame 106 and the current frame 106 can be used to drive the calculation of the target size value based on the notion that the greater the difference between the re-projection transforms, the more of the display panel exhibits the unwanted visual artifacts (e.g., strobing flash between black pixels and illuminated/valid pixels).
- the process 500 may follow the “NO” route from block 504 to block 510 .
- this determination at block 504 may be a determination that re-projection has ceased. For instance, after rendering a frame 106 and proceeding to process a next frame at 502 , the logic of the HMD 100 may determine that the series of frames 106 are being rendered without re-projection (e.g., without any re-projected frames 106 (R)), indicating that the use of re-projection has ceased.
- one or more future frames to be rendered on the display panel 102 may not include any re-projected frames 102 (R), indicating that the use of re-projection has ceased, or is otherwise not occurring at present. This may occur when the application hits frame rate after previously failing to hit frame rate.
- R re-projected frames 102
- logic of the HMD 100 may set the target size values for individual portions of the panel mask 110 to zero (or a minimum size value). For instance, four target size values corresponding to left, top, right, and bottom portions of the panel mask 110 may be set to zero at block 510 .
- logic of the HMD 100 may determine, for each individual portion of the panel mask 110 , whether the target size value is greater than an existing size value for the portion of the panel mask 110 . If it is determined, at block 512 , that the target size value is greater than the existing size value, the process 500 may follow the “YES” route from block 512 to block 514 .
- logic of the HMD 100 may adjust (or change) a size value associated with the portion of the panel mask 110 from the existing size value to the target size value to increase a size of the portion of the panel mask 110 from a current size to an increased size. As indicated, this adjustment to increase the size may be based on the target size value being greater than the existing size value, indicating that the portion of the panel mask 110 is to be brought inward to cover more of the image area 108 . In some embodiments, the size adjustment at block 514 is temporally smoothed to mitigate a sudden field of view change.
- logic of the HMD 100 may render the frame 106 on the display panel(s) 102 along with the panel mask(s) 110 rendered at a periphery of the display panel(s) 102 , wherein at least the portion of the panel mask is rendered at a size in accordance with the target size value, and at block 518 , the next frame 106 , in the series of frames 106 , may be processed by iterating from block 502 for the next frame.
- the panel mask 110 may be rendered at block 516 at an increased size due to the increase from the existing size value to the target size value.
- the algorithm of FIG. 5 may be done independently for each panel mask 110 (L) and 110 (R).
- the same portions of each mask may be adjusted in the same, or a similar, manner in response to particular HMD 100 rotation during re-projection, such as when the leading edge portion of the panel mask 110 on each display panel 102 is expanded inward toward the center of each display panel 102 , as depicted in the examples of FIGS. 1A and 1B .
- the frame 106 for which the panel mask 110 is rendered in accordance with newly determined target size values may be an actual frame 106 (A) or a re-projected frame 106 (R).
- the determination, at block 504 , that re-projection is occurring is based on a previously-rendered re-projected frame 106 (R).
- the subsequent frame may be an actual frame 106 (A) received from the application, in which case, the panel mask 110 is rendered at block 516 in accordance with the calculated target size values along with the actual frame 106 (A).
- the first frame 106 that is rendered with a panel mask 110 that has been adjusted in size may be the next frame, which can be an actual frame 106 (A) received from the application.
- the first frame 106 that is rendered with the increased size panel mask 110 may be the next re-projected frame 106 (R).
- the process 500 may follow the “NO” route from block 512 to block 520 .
- logic of the HMD 100 may determine whether the target size value (for an individual portion of the panel mask 110 ) is less than the existing size value. If the answer at block 520 is “no”, the process 500 may follow the “NO” route from block 520 to block 522 , based on the determination that the target size value is equal to the existing size value.
- the process 500 may follow the “YES” route from block 520 to block 524 .
- logic of the HMD 100 may determine whether a predetermined number of frames 106 (e.g., N frames, where N is equal to 5, 10, etc.) have been consecutively rendered without increasing a size of the relevant portion of the panel mask 110 . If there has been an increase in the size of the relevant portion of the panel mask 110 over the course of the last N consecutive frames 106 , the process 500 may follow the “NO” route from block 524 to block 522 , where no adjustment is made to that portion of the panel mask 110 , and to block 516 where the frame is rendered without any adjustment to the size of the relevant portion of the panel mask 110 .
- a predetermined number of frames 106 e.g., N frames, where N is equal to 5, 10, etc.
- This confidence can be based on a particular number of consecutive frames 106 (A) being rendered without increasing the size of the relevant portion of the panel mask 110 . Accordingly, if there hasn't been any increase to the size of the relevant portion of the panel mask 110 over the course of the last N consecutive frames 106 , the process 500 may follow the “YES” route from block 524 to block 526 .
- the size value may be decremented (e.g., incrementally adjusted downward) towards a lesser target value (e.g., the target size value, or otherwise to a minimum size value of “0”, corresponding to a fully contracted state of the panel mask 110 ).
- a lesser target value e.g., the target size value, or otherwise to a minimum size value of “0”, corresponding to a fully contracted state of the panel mask 110 .
- This may allow for decrementing the size of the portion of the portion of the panel mask 110 over a time period (e.g., a period of roughly two seconds) from the existing size value to the target size value that is less than the existing size value, causing the portion of the panel mask 110 to decrease in size over the period of time from the current size to a lesser size.
- the decrementing at block 526 will eventually cause a previously expanded portion of the panel mask 110 to decrease in size.
- the decrementing operation at block 526 is a slow interpolation step that decrements the existing size value to a lesser target value by a relatively small amount (e.g., a per-frame lerp value of 0.005).
- the logic may be configured to avoid contracting portions of the panel mask 110 faster than a maximum rate of contraction, meaning that if the user's 104 head rotation slows suddenly by a large amount, instead of decreasing the size of the portion of the panel mask 110 all the way down to the determined target size value, the existing size value can, instead, be decremented to an intermediate size value that is greater than the target size value and less than the existing size value. In some embodiments, if the target size value is between the lesser size value and the existing size value, the decrementing at block 526 decrements to the target size value.
- the process 500 may proceed from block 526 to block 516 where the frame 106 is rendered on the display panel 102 along with the panel mask, wherein the portion of the panel mask 110 that was decremented in size is rendered at a decreased size, relative to a previous size of the portion of the panel mask 110 .
- the existing size value e.g., making the existing size value a particular percentage (e.g., 5%) smaller
- the appropriate portions of the panel mask 110 increase in size during re-projection coupled with more HMD 100 rotation, and when HMD 100 rotation slows and/or re-projection ceases, the panel mask 110 portions may decrease in size to reveal more image area 108 on the display panel 102 .
- FIG. 6 illustrates example components of a HMD 600 , such as a VR headset, according to the embodiments disclosed herein may be embedded.
- the HMD 600 may be the same as, or similar to, the HMD 100 referenced in the previous figures, and, therefore, the components of the HMD 600 shown in FIG. 6 may be implemented in the HMD 100 .
- the HMD 600 may be implemented as a standalone device that is to be worn by a user 104 (e.g., on a head of the user 104 ).
- the HMD 600 may be head-mountable, such as by allowing a user 104 to secure the HMD 600 on his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of a user 104 .
- the HMD 600 comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display(s).
- VR virtual reality
- AR augmented reality
- the terms “wearable device”, “wearable electronic device”, “VR headset”, “AR headset”, and “head-mounted display (HMD)” may be used interchangeably herein to refer to the device 600 of FIG. 6 .
- these types of devices are merely example of a HMD 600 , and it is to be appreciated that the HMD 600 may be implemented in a variety of other form factors.
- the HMD 600 includes one or more processors 602 and memory 604 (e.g., computer-readable media 604 ).
- the processors(s) 602 may include a central processing unit (CPU), a graphics processing unit (GPU), both CPU and GPU, a microprocessor, a digital signal processor or other processing units or components known in the art.
- the functionally described herein can be performed, at least in part, by one or more hardware logic components.
- illustrative types of hardware logic components include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.
- FPGAs field-programmable gate arrays
- ASICs application-specific integrated circuits
- ASSPs application-specific standard products
- SOCs system-on-a-chip systems
- CPLDs complex programmable logic devices
- each of the processor(s) 602 may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.
- the memory 604 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data.
- Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
- the memory 604 may be implemented as computer-readable storage media (“CRSM”), which may be any available physical media accessible by the processor(s) 602 to execute instructions stored on the memory 602 .
- CRSM may include random access memory (“RAM”) and Flash memory.
- RAM random access memory
- CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) 602 .
- the HMD 600 may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein.
- the computer-readable media 604 is shown as including various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) 602 for carrying out the techniques, functionality, and/or operations described herein.
- a few example functional modules are shown as stored in the computer-readable media 604 and executable on the processor(s) 602 , although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic.
- SOC system on a chip
- An operating system module 606 may be configured to manage hardware within and coupled to the HMD 600 for the benefit of other modules.
- the HMD 600 may include one or more applications 608 stored in the memory 604 or otherwise accessible to the HMD 600 .
- the application(s) 608 includes a gaming application 610 .
- the HMD 600 may include any number or type of applications and is not limited to the specific example shown here.
- the gaming application 610 may be configured to initiate gameplay of a video-based, interactive game (e.g., a VR game) that is playable by the user 104 , and to output frames (e.g., actual frames 106 (A)) to be rendered on the display panels of the HMD 100 .
- a video-based, interactive game e.g., a VR game
- frames e.g., actual frames 106 (A)
- a compositor 611 may be configured to render, and to dynamically adjust, a panel mask 110 along with each frame as a late-stage rendering operation. Accordingly, the compositor 611 , along with associated logic of the HMD 100 , may perform the dynamic panel mask adjustment techniques disclosed herein.
- the HMD 600 has input devices 612 and output devices 614 .
- the input devices 612 may include control buttons.
- one or more microphones may function as input devices 612 to receive audio input, such as user voice input.
- one or more cameras or other types of sensors e.g., inertial measurement unit (IMU)
- IMU inertial measurement unit
- additional input devices 612 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like.
- the HMD 600 may omit a keyboard, keypad, or other similar forms of mechanical input.
- the HMD 600 may be implemented relatively simplistic forms of input device 612 , a network interface (wireless or wire-based), power, and processing/memory capabilities.
- a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that the HMD 600 can thereafter be used.
- the input device(s) 612 may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons.
- the output devices 614 may include a display(s) 616 , which may be the same as, or similar to, the display panel(s) 102 described with reference to the previous figures.
- the output devices 614 may further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like.
- a simple light element e.g., LED
- LED to indicate a state such as, for example, when power is on.
- the HMD 600 may further include a wireless unit 618 coupled to an antenna 620 to facilitate a wireless connection to a network.
- the wireless unit 618 may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the HMD 600 may further include physical ports to facilitate a wired connection to a network, a connected peripheral device (including a PC, game console, etc.), or a plug-in network device that communicates with other wireless networks.
- the HMD 600 may further include optical subsystem 622 that directs light from the electronic display(s) 616 to a user's eye(s) using one or more optical elements.
- the optical subsystem 622 may include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth.
- one or more optical elements in optical subsystem 622 may have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optical subsystem 622 allows electronic display(s) 616 to be physically smaller, weigh less, and consume less power than larger displays.
- magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images).
- FOV field of view
- the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., 120-150 degrees diagonal), and in some cases all, of the user's FOV.
- AR applications may have a narrower FOV (e.g., about 40 degrees FOV).
- Optical subsystem 622 may be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth.
- content provided to electronic display(s) 616 for display is pre-distorted, and optical subsystem 622 corrects the distortion when it receives image light from electronic display(s) 616 generated based on the content.
- the HMD 600 may further include one or more sensors 624 , such as sensors used to generate motion, position, and orientation data.
- sensors 624 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors,
- the sensors 624 may also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data.
- a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data.
- the HMD 600 may include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of the HMD 600 .
- the sensor(s) 624 may include an inertial measurement unit (IMU) 626 .
- IMU 626 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated with IMU 626 , or sonic combination thereof Based on the measurement signals such motion-based sensors, such as the IMU 626 , may generate calibration data indicating an estimated position of HMD 600 relative to an initial position of HMD 600 .
- multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll).
- IMU 626 can, for example, rapidly sample the measurement signals and calculate the estimated position of HMD 600 from the sampled data.
- IMU 626 may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD 600 .
- the reference point is a point that may be used to describe the position of wearable device 702 . While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD 600 (e.g., a center of the IMU 626 ), Alternatively, IMU 626 provides the sampled measurement signals to an external console (or other computing device), which determines the calibration data.
- the sensors 624 may operate at relatively high frequencies in order to provide sensor data at a high rate.
- sensor data may be generated at a rate of 1000 Hz (or 1 sensor reading every 1 millisecond). In this way, one thousand readings are taken per second.
- the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milliseconds.
- the sensors 624 may include light sensors that are sensitive to light emitted by base stations in the environment of the HMD 600 for purposes of tracking position and/or orientation, pose, etc., of the HMD 600 in 3D space.
- the calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by the sensors 624 .
- the HMD 600 may further include an eve tracking module 628 .
- a camera or other optical sensor inside HMD 600 may capture image information of a user's eyes, and eye tracking module 628 may use the captured information to determine interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to HMD 600 (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye.
- infrared light is emitted within HMD 600 and reflected from each eye. The reflected light is received or detected by a camera of the HMD 600 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye.
- eye tracking module 628 may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a user 104 to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking).
- eye tracking module 628 may integrate information from past measurements, measurements identifying a position of a user's 104 head, and 3 D information describing a scene presented by electronic display(s) 616 .
- information for the position and orientation of the user's 104 eyes is used to determine the gaze point in a virtual scene presented by HMD 600 where the user 104 is looking.
- the HMD 600 may further include a head tracking module 630 .
- the head tracking module 630 may leverage one or more of the sensor 624 to track head motion, including head rotation, of the user 104 , as described above.
- the head tracking module 630 can track up to six degrees of freedom of the HMD 600 (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at every frame 106 of a series of frames 106 so that the application 608 can determine how to render a scene in the next frame 106 (even for re-projected frames 106 (R)) in accordance with the head position and orientation.
- the head tracking module 630 is configured to predict a future position and/or orientation of the HMD 600 based on current and/or past data. This is because the application is asked to render a frame 106 before the user 104 actually sees the light (and, hence, the image) on the display(s) 616 . Accordingly, a next frame 106 can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time, such as roughly 25-30 milliseconds (ms) prior to rendering the frame 106 .
- Rotation data provided by the head tracking module 630 can be used to determine both direction of HMD 600 rotation, and amount of HMD 600 rotation in any suitable unit of measurement.
- rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down.
- Amount of rotation may be in terms of degrees, radians, etc.
- Angular velocity may be calculated to determine a rate of rotation of the HMD 600 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Optics & Photonics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Controls And Circuits For Display Device (AREA)
- Control Of Indicators Other Than Cathode Ray Tubes (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
- Virtual reality (VR) systems are used both within and outside of the video game industry. Displays for VR systems, such as those embedded in a VR headset, typically operate at a minimum refresh rate that is suitable for VR applications. For instance, 90 Hertz (Hz) is a common refresh rate for VR displays. In a “live rendering” scenario, a graphics-based application, such as a video game, outputs frames for rendering at a frame rate that matches the refresh rate of the display, meaning that a new frame received from the application (referred to herein as an “actual frame”) is displayed at every screen refresh. Such a live rendering scenario is often referred to as the application “hitting frame rate.”
- In practice, an application does not always hit frame rate for various reasons. For example, the application may intermittently drop a frame, and/or the application may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). In situations where the application is not hitting frame rate, a technique called “rotation-only re-projection” can be used to replace missing frames with re-projected frames in a way that accounts for the user's head rotation, making it appear to the user as if the application is hitting frame rate. Without re-projection, for example, a deficient frame rate from the application may cause in-game stuttering or hitching. In VR applications, where the user is fully immersed in the virtual environment, the user can become nauseous if the application fails to hit frame rate and there is no re-projection to compensate for the missing frames. Thus, re-projection is a technique that allows for a better user experience when an application is not hitting frame rate. Consider an example where the application is outputting frames at half the ideal frame rate (e.g., 45 frames per second where 90 frames per second is the ideal frame rate). In this example, every other frame can be re-projected using pixel data from the most recently-rendered actual frame to create a re-projected frame that transforms the scene (e.g., through rotation and re-projection calculations) to match the re-projected scene to the user's current head orientation. This makes it look to the user as if the scene is moving in a way that is expected given the user's head rotation, even when re-projected frames are used to compensate for the application failing to hit frame rate.
- Although rotation-only re-projection prevents in-game stuttering or hitching, it produces its own unwanted visual artifacts during head rotation, at least in VR systems that use low-persistence displays (e.g., where the display is illuminated for a small fraction of the frame time). For example, rotation-only re-projection can cause a strobing flash to appear at the leading edge of each display panel in the direction of head rotation when the user is rotating his/her head. This strobing flash occurs during head rotation because the pixels at the leading edge of the display panel toggle every frame between an actual frame's valid pixels and solid black pixels in the following re-projected frame. The black pixels occur during head rotation in the re-projected frames because there is no pixel data available from the previously-rendered actual frame to use for the pixels at the leading edge of the display panel for the re-projected frame. Accordingly, when the user rotates his/her head while re-projection is being used, the leading edges of the display panels in the direction of the user's head rotation toggles every frame between black pixels and illuminated pixels, causing an unwanted strobing artifact at the leading edges of the display panels.
- Provided herein are technical solutions to improve and enhance these and other systems.
- The detailed description is described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.
-
FIG. 1A is a diagram illustrating example left and right display panels of a head-mounted display (HMD) where a frame is rendered with a dynamically expanded panel mask during head rotation while frames are being processed using re-projection, the direction of head rotation being in a rightward (or positive X) direction. -
FIG. 1B is a diagram illustrating the example left and right display panels of the HMD ofFIG. 1A where a frame is rendered with a dynamically expanded panel mask during head rotation while frames are being processed using re-projection, the direction of head rotation being in a leftward (or negative X) direction. -
FIG. 2 is a diagram illustrating example left and right display panels of a HMD showing a difference between a fully contracted state of the panel masks and a fully expanded state of the panel masks. -
FIG. 3 is another diagram illustrating example left and right display panels of a HMD showing the difference between a fully contracted state of the panel masks and a fully expanded state of the panel masks. -
FIG. 4 is a flow diagram of an example process for dynamically re-sizing a portion of a panel mask on a display panel of a HMD, in accordance with embodiments disclosed herein. -
FIG. 5 is a flow diagram of a more detailed example process for dynamically re-sizing a portion of a panel mask on a display panel of a HMD, in accordance with embodiments disclosed herein. -
FIG. 6 illustrates example components of a wearable device, such as a VR headset, in which the techniques disclosed herein can be implemented. - Described herein are, among other things, techniques for dynamically adjusting (increasing and decreasing) a size of a panel mask that is rendered at a periphery of a display panel of a head-mounted display (HMD) in order to hide unwanted visual artifacts from view, as needed. A HMD may be worn by a user for purposes of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment. One or more display panels of the HMD render images based on frames that are output by an application (e.g., a video game), and these images are viewed by a user through the optics that are included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment.
- A panel mask(s) may be rendered at a periphery of the display panel(s) of the HMD to provide a border around the images presented on the display panel(s). The panel mask frames the image much like a picture frame. In some embodiments, the inner edges of the panel mask(s) can be blurred (e.g., using a subtle vignette) to make the panel mask appear to the user as a near-field object that is close to (or disposed on) on the user's face, much like facial features (e.g., the nose bridge between the eyes) are seen when focusing on far-field objects.
- As mentioned, the HMD may utilize a technique called “re-projection” to compensate for the application not hitting frame rate. During re-projection, a user wearing the HMD may rotate his/her head, which, as mentioned, may cause an unwanted strobing artifact to appear at the leading edge of the display panel(s) that is presently not covered by the panel mask. Accordingly, the HMD can detect when re-projection is, or is not, being used, and at least a portion of the panel mask can be dynamically adjusted in size based upon whether re-projection is, or is not, being used. For example, in cases where re-projection is ongoing, a portion of the panel mask may be adjusted (e.g., contracted away from a center of the display panel or expanded towards the center of the display panel) by an appropriate amount depending on an amount of rotation of the HMD during re-projection. This allows for covering up (or hiding) unwanted visual artifacts, such as a strobing flash, that may manifest during re-projection coupled with head movement. For instance, if the user rotates his/her head in a rightward direction while frames are being re-projected, the portion of the panel mask at the leading (e.g., right) edge of the display panel(s) may be increased in size to ensure that the unwanted strobing artifact is covered by the panel mask so that the user does not see the strobing flash in the image area of the display panel. When the user subsequently slows or ceases head rotation, or when the application hits frame rate again (i.e., the use of re-projection ceases), this expanded portion of the panel mask can be adjusted in size over time back to a fully contracted state so that more of the scene is rendered in the image area of the display panel. This technique of dynamically expanding and contracting at least a portion of the panel mask, based on head rotation coupled with re-projection, allows for a better viewing experience for a user of the HMD because the user does not notice the unwanted strobing artifact caused by re-projection during head rotation; the panel mask expands to occlude the strobing artifact when it is present, and the panel mask contracts after the strobing artifact has ceased to reveal more of the scene in the image area of the display panel.
- An example process for dynamically adjusting a size of a portion of a panel mask, according to the embodiments described herein, includes determining, by one or more processors, whether a series of frames are being rendered on a display panel of a HMD using re-projection, determining, based at least in part on whether the series of frames are being rendered using re-projection, a target size value for at least a portion of a panel mask, determining that a size value associated with at least the portion of the panel mask is set to an existing size value that is not equal to the target size value, adjusting the size value from the existing size value to the target size value to increase or decrease a size of at least the portion of the panel mask from a current size to an adjusted size, and rendering a frame, of the series of frames, on the display panel along with the panel mask rendered at a periphery of the display panel, wherein at least the portion of the panel mask is rendered at the adjusted size in accordance with the target size value. If the size adjustment is to increase the size of the portion of the panel mask, the size of the portion of the panel mask can be subsequently decreased in size if the use of re-projection ceases and/or if head rotation slows or ceases altogether. For example, if the use of re-projection ceases, the portion of the panel mask can be decreased in size incrementally over a time period to a minimum size or some other target value. In this way, the contraction of the panel mask back to its fully contracted state is not visually distracting to the user.
- Also disclosed herein are systems, including a HMD, configured to implement the techniques and processes disclosed herein, as well as non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications where HMDs are used, including, without limitation, non-VR applications (e.g., AR applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.
-
FIG. 1A is a diagram illustrating an example head-mounted display (HMD) 100 that includes a left display panel 102(L) and a right display panel 102(R), which make up a pair of display panels. Although the example HMD 100 includes two display panels, 102(L) and 102(R), it is to be appreciated that the HMD 100 may include asingle display panel 102, or more than twodisplay panels 102. Hence, “display panel 102,” as used herein, may refer to either of the display panels 102(L) or 102(R) of a two-panel HMD 100, as shown inFIG. 1A , or it may refer to asingle display panel 102 of a HMD 100 with any number of display panels (e.g., a single-panel HMD 100 or a multi-panel HMD 100). - The display panel(s) 102 may represent any suitable type of display, such as an emissive display that utilizes light emitting elements to emit light during presentation of image frames (herein referred to as “frames”) on the display panel(s) 102. As an example, the left and right display panels 102(L) and 102(R) may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display that utilizes a suitable display technology for HMD applications.
- The HMD 100 may represent a VR headset for use in VR systems, such as for use with a VR gaming system. However, the HMD 100 may additionally, or alternatively, be implemented as an AR headset for use in AR applications. In AR, a user sees virtual objects overlaid on a real-world environment, whereas, in VR, the user does not see a real-world environment, but is fully immersed in a virtual environment, as perceived via the
display panels 102 and the optics (e.g., lenses) of theHMD 100. Examples described herein pertain primarily to a VR-basedHMD 100, but it is to be appreciated that theHMD 100 is not limited to implementation in VR applications. - In general, an application executing on a computing device—such as the
HMD 100 itself, or a computing device (e.g., a personal computer (PC), game console, etc.) associated with, and coupled to, theHMD 100—may be configured to output a series offrames 106 that are ultimately rendered on the display panel(s) 102 of theHMD 100. The image data of theframes 106 is presented within animage area 108 of the display panel(s) 102 that is not covered by apanel mask 110.FIG. 1A shows a left image area 108(L) at the center of the left display panel 102(L) and inside a left panel mask 110(L), and a right image area 108(R) at the center of the right display panel 102(R) and inside a right panel mask 110(R). - The series of
frames 106 may be output from a video game application, or any other type of graphics-based application. The application may be executed in a graphics pipeline that outputs per-pixel values (e.g., color values) that are combined to create on image (within the image area 108) on the display panel(s) 102 with a desired visual effect. During runtime, a head tracking module of theHMD 100 may generate data about the position and pose of theHMD 100 that is provided to the application in order to inform the application regarding how to render anext frame 106 in the series offrames 106 in accordance with the user's 104 head position/pose. This allows for rendering imagery within the image area(s) 108 on the display panel(s) 102 in a way that makes theuser 104 believe he/she is looking around a virtual environment that includes objects (both static and moving objects), and these objects move within the scene in an expected manner along with the user's 104 head movement. - As mentioned, a left panel mask 110(L) is rendered at a periphery of the left display panel 102(L) and a right panel mask 110(R) is rendered at a periphery of the right display panel 102(R). As used herein, “
panel mask 110” may refer to either of the panel masks 110(L) or 110(R), as shown inFIG. 1A . However, a single-panel HMD 100 may include asingle panel mask 110 that surrounds theimage area 108 of asingle display panel 102. At startup, theHMD 100 may compute a mesh for the panel mask(s) 110, which is used to render the panel mask(s) 110 in a fully contracted state at the periphery of the display panel(s) 102, meaning that the portions of the panel mask(s) 110 that are dynamically adjustable in size are initially rendered at a smallest size to reveal more of the scene within the image area(s) 108. In general, the panel mask(s) 110 may be persistently rendered along with each frame, although the size of individual portions of the panel mask(s) 110 may be adjusted in particular ones of the frames to hide unwanted visual artifacts, as needed. In some embodiments, the panel mask(s) 110 may be implemented as a plurality of black pixels at the periphery of the display panel(s) 102, which may be added to the frame buffer before rendering a frame. Although the panel mask(s) 110 is depicted in the figures as a generally annular-shaped panel mask that encloses a generally circular-shapedimage area 108, it is to be appreciated that other geometries can be used to implement the panel mask(s) 110, such as by rendering the panel mask(s) 110 as a square, rectangular, or other geometrically-shaped frame surrounding theimage area 108. In some embodiments, the panel mask(s) 110 may be implemented as three separate panel masks corresponding to red, green, and blue color channels, wherein eachseparate panel mask 110 of the threepanel masks 110 is shaped based on a known chromatic distortion of the optics included in theHMD 100 to mitigate effects of color fringing, which may occur at or near the outer edge of thedisplay panel 102. When theuser 104 looks through the lenses in theHMD 100, for example, there may be some natural spatial (radial) distortion due to the lenses distorting the image to some degree (known as a “pincushion” effect in VR headsets). There may also be some chromatic distortion (e.g., red, green, blue (RGB) separation). If the user's 104 eyes happen to be close enough to the lens assembly to where theuser 104 can actually see the edges of thedisplay panels 102, theuser 102 may see cyan fringing on the panel edges, which comes from the chromatic distortion that separates RGB, where the red pixels are chromatically distorted the most (toward the center of the display panel 102), followed by a lesser amount of distortion for the green pixels, and the least amount of distortion for the blue pixels (e.g., the blue is distorted the least, remaining closest to the edge of the panel, causing the cyan fringing effect). This color fringing can be quite distracting for the users who can see it (depending on the thickness of the foam used in theHMD 100, the spatial geometry of the user's face, and so on, theuser 104 may or may not see the color fringing). Accordingly, the separate panel masks of a three-panel mask implementation on eachdisplay panel 102 may independently mask red, green, and blue to mitigate the color fringing effect. -
FIG. 1A shows the left panel mask 110(L) as a relatively dark grey area, and the right panel mask 110(R) as a relatively light grey area. This is merely for purposes of distinguishing the two panel masks from each other in the relevant figures, and it is to be appreciated that the pair of panel masks 110(L) and 110(R) may be rendered in a uniform color (e.g., black pixels) on both display panels 102(L) and 102(R). In this manner, theuser 104 perceives the two panel masks 110(L) and 110(R) as a single panel mask that surrounds the two image areas 108(L) and 108(R), much like a human perceives an environment with eye sockets and a nose bridge that surround each eye in a near field of view. As mentioned, theinner edge 112 of each panel mask 110(L) and 110(R) may be blurred slightly (e.g., using a subtle vignette) to mimic near-field facial features, which is more familiar to humans than a sharpinner edge 112 would be. - In addition, individual portions of each of the panel masks 110(L) and 110(R) are dynamically adjustable in size to expand and contract portions of the
panel mask 110 to cover more or less of theimage area 108, as needed, for eachframe 106 of the series offrames 106. For example, a portion of the panel mask(s) 110 may increase in size by expanding inward toward a center of the display panel(s) 102, which allows for covering up (or hiding) unwanted visual artifacts that manifest at that portion of the display panel(s) 102 during re-projection coupled with head rotation. - As mentioned, displays used in VR systems typically operate at a minimum refresh rate that is suitable for VR applications. As such, the display panel(s) 102 of the
HMD 100 may operate at a refresh rate of say, 90 Hz, which is a common refresh rate for VR displays. Although 90 Hz is used as an example refresh rate, it is to be appreciated that the display panel(s) 102 may operate at other, different refresh rates, without changing the basic characteristics of the techniques and systems disclosed herein. In a “live rendering” scenario, the application that outputs the series offrames 106 may output actual frames 106(A) at a frame rate that matches the refresh rate of the display panel(s) 102 (e.g., actual frames 106(A) are output at 90 frames per second). However, in instances where the application is not hitting frame rate (i.e., outputting actual frames 106(A) at something less than 90 frames per second), theHMD 100 may be configured to employ re-projection (e.g., rotation-only re-projection) to replace missing actual frames 106(A) with re-projected frames 106(R), and theHMD 100 may do so in a way that accounts for the user's 104 head rotation, making it appear to theuser 104 as if the application is hitting frame rate and that the scene is moving about thedisplay panels 102 in an expected manner given the user's 104 head rotation.FIG. 1A illustrates this re-projection scenario—where the series offrames 106 includes an actual frame 106(A)(1), followed by a re-projected frame 106(R)(2), followed by another actual frame 106(A)(3), and so on and so forth. For instance, in the example ofFIG. 1A , the application may be outputting frames at half of the ideal frame rate (e.g., at a frame rate of 45 frames per second, where 90 frames per second is the ideal frame rate). In this case, every other frame between sequential actual frames 106(A) is a re-projected frame 106(R). A re-projected frame 106(R) is generated using pixel data from a recently-rendered actual frame 106(A) (e.g., the most recently-rendered actual frame 106(A)) to create a scene that is transformed (e.g., through rotation and re-projection calculations) in a way that accounts for the user's 104 head rotation. Assuming that the actual frame 106(A)(1) is rendered before the re-projected frame 106(R)(2) inFIG. 1A , the re-projected frame 106(R)(2) can be derived from the pixel data associated with the previously-rendered actual frame 106(A)(1). In this case, the scene of the previously-rendered actual frame 106(A)(1) is rotated and re-projected to create the re-projected frame 106(R)(2) in a manner that theuser 104 would expect given his/her head movement. - When the series of
frames 106 are being rendered on the display panel(s) 102 of theHMD 100 using re-projection, and when theuser 104 is rotating his/her head, a strobing flash may appear at the leading edge of eachdisplay panel 102 in the direction of head rotation. For instance, inFIG. 1A , theuser 104 is shown as rotating his/her head in a rightward (i.e., positive X) direction of rotation. In this scenario, the aforementioned strobing flash may occur during re-projection, and it may manifest at the right edge (i.e., the leading edge in the direction of rotation) of each display panel 102(L) and 102(R) because there is no pixel data available in the previously-rendered actual frame 106(A)(1) to render the pixels at the right edge of the each display panel 102(L) and 102(R) for the re-projected frame 106(R)(2) since theuser 106 has rotated in the rightward direction. Accordingly, theHMD 100 may be configured to dynamically increase the size of a right portion of eachpanel mask 110 to cover up (or hide) this unwanted visual artifact. Without changing the size of the right portion of eachpanel mask 110, this strobing artifact may be visible within the image area(s) 108 of the display panel(s) 102. - In an illustrative example, the
HMD 100 may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to determine whether a series offrames 106 are being rendered on a display panel(s) 102 of theHMD 100 using re-projection. That is, the logic may determine whether one or more of theframes 106 in the series offrames 106 are re-projected frames 106(R) generated from pixel data associated with actual frames 106(A) received from an application that have already been rendered on the display panel(s) 102. In some embodiments, the logic may make this determination upon rendering the re-projected frame 106(R)(2). In other embodiments, the logic may make this determination after rendering the actual frame 106(A)(1), but before rendering the re-projected frame 106(R)(2). In any case, if re-projection is being used, the logic may also be configured to determine, based at least in part on rotation data provided by a head tracking module of theHMD 100 indicating that theHMD 100 has rotated (e.g., between time, t1, and time, t2), a target size value for each adjustable portion of the panel mask(s) 110. If the target size value for a given portion of the panel mask(s) 110 is not equal to an existing size value for that portion, the logic may respond by adjusting a size value associated with at least a portion of the panel mask(s) 110 from the existing size value to the target size value to increase a size of at least the portion of thepanel mask 110 from a current size to an adjusted size, and by rendering a frame (e.g., the actual frame 106(A)(3)), of the series offrames 106, on the display panel(s) 102 along with the panel mask(s) 110 rendered at a periphery of the display panel(s) 102, wherein at least the portion of the panel mask(s) 110 is rendered at the adjusted size. Consider an example where the left panel mask 110(L) and the right panel mask 110(R) are each partitioned into left, right, top, and bottom portions.FIG. 1A shows that a right portion of the left panel mask 110(L) and a right portion of the right panel mask 110(R)—each portion corresponding to the rightward direction of rotation—increase in size from a current size to an increased size. This is indicated inFIG. 1A by the inner edge 112(L) of the left panel mask 110(L) moving inward toward the center of the left display panel 102(L) (i.e., from the dashed line to the solid line), and by the inner edge 112(R) of the right panel mask 110(R) moving inward towards the center of the right display panel 102(R) (i.e., from the dashed line to the solid line). In other words, the right portion of eachpanel mask 110 is pulled inward to expand the size of the right portion of eachpanel mask 110 so that more of theimage area 108 on the right edge/side of the display panel(s) 102 is covered by the panel mask(s) 110. These “inward” adjustments occur when the target size value is greater than the existing size value for the relevant portion of the panel mask(s) 110. -
FIG. 1B is a diagram illustrating the example left andright display panels 102 of theHMD 100 ofFIG. 1A , where a frame 106 (e.g., the actual frame 106(A)(3)) is rendered with a dynamically expanded panel mask(s) 110 during head rotation whileframes 106 are being processed using re-projection, the direction of head rotation being in a leftward (or negative X) direction inFIG. 1B , which is opposite to the direction of rotation shown inFIG. 1A . In the scenario ofFIG. 1B , the logic of theHMD 100 is configured to respond similarly to how it responds in the scenario ofFIG. 1A , the difference being that a different portion (i.e., the left portion) of eachpanel mask 110 is resized in response to the different direction of rotation that is detected by theHMD 100. For example, when there is leftward (i.e., negative X) rotation during re-projection, the left portion of eachpanel mask 110 is pulled inward to expand the size of the left portion of thepanel mask 110 so that more of theimage area 108 on the left edge/side of the display panel(s) 102 is covered by the panel mask(s) 110. - Likewise, it can be appreciated that upward or downward directions of head rotation can cause the top portion and bottom portion, respectively, of each
panel mask 110 to dynamically increase in size (e.g., by expanding inward toward the center of the display panel(s) 102) so that more of theimage area 108 on the top/bottom edge/side of the display panel(s) 102 is covered by the panel mask(s) 110. It can be appreciated that, if a rotational directionality of the HMD's 100 is a vector having both X and Y components, this may cause adjacent portions of thepanel mask 110 to simultaneously expand in size. For example, if theuser 104 were to rotate his/her head diagonally into the positive X-Y quadrant of the reference coordinate system shown inFIGS. 1A and 1B , both the right portion and the top portion of the panel mask(s) 110 may be simultaneously expanded in size such that the two adjacent portions of thepanel mask 110 expand inward toward the center of thedisplay panel 102 at the same time. It can also be appreciated that, in some implementations, the entire panel mask(s) 110 (i.e., all portions) may be expanded in size, causing the image area(s) 108 to contract (or get smaller) as more of the image area(s) 108 is covered by an expanded panel mask(s) 110. Subsequently, when the entire panel mask(s) 110 contracts down to a smaller size, the image area(s) 108 may dilate back to a maximum size of the image area(s) 108 as less of the image area(s) 108 is covered by a contracted panel mask(s) 110. -
FIG. 2 is a diagram illustrating example left andright display panels 102 of aHMD 100 showing a difference between a fully contracted state of a panel mask(s) 110 at the top ofFIG. 2 and a fully expanded state of the panel mask(s) 110 at the bottom ofFIG. 2 . As mentioned, individual portions of thepanel mask 110 for eachdisplay panel 102 can be dynamically adjusted in size, as needed, to cover more or less of theimage area 108 of eachdisplay panel 102. There may be limits or bounds for these size adjustments. For example, an individual portion of thepanel mask 110 may be increased up to, but not beyond, a maximum size, and decreased down to, but not below, a minimum size. In this sense, thepanel mask 110 can be persistently rendered with each frame, and its size is controllable based on values that range between the minimum size value (e.g., “0”) and the maximum size value (e.g., “1”). In other words, the size value for an individual portion of thepanel mask 110 is clamped between a minimum size value (e.g., “0”) and a maximum size value (e.g., “1”), but is adjustable to any number between these two bounds, inclusive of these two bounds. One way of thinking about it is a percentage adjustment (e.g., 0 is 0%, 0.4 is 40%, 0.6 is 60%, 1 is 100%, etc.). As noted, this size adjustment can be controlled independently for individual portions of thepanel mask 110. - For instance, consider partitioning the left panel mask 110(L) into four portions including a left portion 110(L)(1), a top portion 110(L)(2), a right portion 110(L)(3), and a bottom portion 110(L)(4). Likewise, the right panel mask 110(R) may be partitioned into four portions including a left portion 110(R)(1), a top portion 110(R)(2), a right portion 110(R)(3), and a bottom portion 110(R)(4). Each of these portions of the
panel mask 110 can be adjusted in size between a minimum size value (e.g., a size value set to “0”) corresponding to the fully contracted state at the top ofFIG. 2 , and a maximum size value (e.g., a size value set to “1”) corresponding to the fully expanded state at the bottom ofFIG. 2 . Thus, there may be at least four dynamically adjustable size values for eachpanel mask 110 to control the size of the individual portions of thepanel mask 110. Theinner edge 112 of the panel mask 110 (at each portion of the panel mask 110) can be moved partway between the minimum extent (shown at the top ofFIG. 2 ) and the maximum extent (shown at the bottom ofFIG. 2 ) by setting the size value to a value between the minimum size value and the maximum size value (e.g., a size value between “0” and “1”). There may be reasons for persistently rendering the panel mask(s) 110 at a minimum size (i.e. in the fully contracted state) regardless of whether re-projection is being used and/or head rotation is present. For example, someHMDs 100 may exhibit color fringing at or near the outer edge of thedisplay panel 102, and it may be desirable to cover up (or hide) such visual artifacts so that they are not a distraction to theuser 104. Meanwhile, the panel mask(s) 110 need to expand in size beyond the maximum size value because there are practical limits to the rate of rotation of theHMD 100 when it is worn by auser 104, and there are practical limits to the perceivable visual elements when theuser 104 is rotating his/her head at high speeds. For example, it is practically impossible for theaverage user 104 to rotate his/her head above a particular threshold angular velocity, and, therefore, thepanel mask 110 does not need to be expanded to cover the entirety of thedisplay panel 102 in this regard; the expansion of the panel mask can “bring in” the field of view just enough to cover unwanted visual artifacts. Moreover, even if theuser 104 could rotate his/her head beyond this threshold angular velocity, theuser 104 would not be able to discern the unwanted visual artifacts that manifest themselves at those high speeds of rotation because the eye simply can't track these visual elements when the field of view is changing at a rapid rate. -
FIG. 3 is another diagram illustrating example left and right display panels 102(L) and 102(R) of aHMD 100 showing the difference between a fully contracted state of the panel masks 110(L) and 110(R) and a fully expanded state of the panel masks 110(L) and 110(R). InFIG. 3 , the fully contracted state is shown by the dashed lines that represent where theinner edges 112 of the panel masks 110 would be in the fully contracted state. Each portion (e.g., top, bottom, left, and right) of thepanel mask 110 can have a size value (SV) set to zero (e.g., SV=0) to move theinner edge 112 of that portion of thepanel mask 110 outward away from the center of thedisplay panel 102 such that the portion of thepanel mask 110 is decreased to its minimum size. The individual portions of thepanel mask 110 may have their size values set to zero at startup of theHMD 100. InFIG. 3 , the fully expanded state of the panel mask(s) 110 is shown by the solid lines that represent where theinner edges 112 of the panel masks 110 would be in the fully expanded state. Each portion of thepanel mask 110 can have its size value set to one (e.g., SV=1) to move theinner edge 112 of that portion of thepanel mask 110 inward toward the center of thedisplay panel 102 such that the portion of thepanel mask 110 is increased to its maximum size. As mentioned, the individual portions of thepanel mask 110 can be adjusted to an intermediate size between the minimum size and the maximum size by setting the size value for each portion to an intermediate value between the minimum size value (e.g., “0”) and the maximum size value (e.g., “1”). - The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.
-
FIG. 4 is a flow diagram of anexample process 400 for dynamically re-sizing a portion of apanel mask 110 on adisplay panel 102 of aHMD 100, in accordance with embodiments disclosed herein. For discussion purposes, theprocess 400 is described with reference to the previous figures. - At 402, logic of a
HMD 100 may determine a target size value(s) for at least a portion(s) of a panel mask(s) 110. For a givenframe 106, the logic of theHMD 100 may, for example, determine four target size values corresponding to a left portion 110(L)(1), a top portion 110(L)(2), a right portion 110(L)(3), and a bottom portion 110(L)(4) of a left panel mask 110(L), as well as four target size values corresponding to a left portion 110(R)(1), a top portion 110(R)(2), a right portion 110(R)(3), and a bottom portion 110(R)(4) of a right panel mask 110(R). Thus, a total of eight target size values may be determined atblock 402 for a two-panel HMD 100 that renders a left panel mask 110(L) and aright panel mask 110 for a givenframe 106. The determination of the target size value(s) atblock 402 may depend on whether re-projection is, or is not, currently being used to render the series offrames 106. If re-projection is ongoing (e.g., if the givenframe 106 is part of a series offrames 106 that are being rendered using re-projection), then the target size value(s) may be determined based at least in part on rotation data provided by a head tracking module of theHMD 100. If re-projection is not ongoing, the target size value(s) may be set to a value(s) of zero atblock 402. As mentioned, the target size value for an individual portion of thepanel mask 110 may be clamped between a minimum size value (e.g., “0”) and a maximum size value (e.g., “1”) such that the target size value can be set to any value between these two bounds (e.g., between “0” and “1”), inclusive of these two bounds. In particular, while re-projection is ongoing, the target size value may be calculated as a value that corresponds to an amount that theHMD 100 has rotated. For instance, greater amounts ofHMD 100 rotation correspond to target size values that are closer to the maximum size value (e.g., “1”), and lesser amounts ofHMD 100 rotation correspond to target size values closer to the minimum size value (e.g., “0”). - At 404, the logic of the
HMD 100 may adjust a size value associated with at least the portion of thepanel mask 110 from an existing size value to the target size value determined atblock 402, assuming that the target size value is not equal to the existing size value. That is, if the target size value is equal to the existing size value, then the logic may refrain from making a size adjustment atblock 404. Accordingly, if re-projection is ongoing, and if the logic determines that the size value associated with the portion of the panel mask is set to an existing size value that is less than the target size value, the adjustment atblock 404 may result in a size of at least the portion of thepanel mask 110 increasing from a current size to an increased size. If re-projection is not ongoing (resulting in a target size value of zero at block 402) and if the existing size value is currently non-zero, or if re-projection is ongoing and a non-zero target size value is otherwise less than the existing size value, the adjustment atblock 404 may result in a size of at least the portion of thepanel mask 110 decreasing from a current size to a decreased size. A downward size adjustment to a decreased size atblock 404 may be a slow interpolation from a current size value to a lesser target value (e.g., the minimum value or some other target value) by a small amount (e.g., a per-frame lerp value of 0.005). In some embodiments, a downward size adjustment atblock 404 is further conditioned on having not increased the size of the portion of thepanel mask 110 over the course of a predetermined number ofconsecutive frames 106 preceding thecurrent frame 106. For example, if the size of the portion of thepanel mask 110 has not increased in the last ten consecutive frames, the size may be adjusted downward to a decreased size. Otherwise, if this consecutive frame criterion is not met, there may not be any adjustment to the size value atblock 404. - At 406, the logic of the
HMD 100 may render aframe 106, of the series offrames 106, on thedisplay panel 102 along with thepanel mask 110 rendered at a periphery of thedisplay panel 102, wherein at least the portion of thepanel mask 110 is rendered at a size (e.g., an adjusted size) in accordance with the target size value. - The
process 400 allows for hiding unwanted visual artifacts that may occur during re-projection coupled with head rotation by virtue of thepanel mask 110 expanding to cover the area of thedisplay panel 102 where the unwanted visual artifacts would otherwise present themselves. This is shown, by way of example and not limitation, inFIGS. 1A and 1B , where a portion of thepanel mask 110 on eachdisplay panel 102 is increased in size from a current size to an increased size and rendered with aframe 106 on the display panel(s) 102. -
FIG. 5 is a flow diagram of a moredetailed example process 500 for dynamically re-sizing a portion of apanel mask 110 on adisplay panel 102 of aHMD 100, in accordance with embodiments disclosed herein. For discussion purposes, theprocess 500 is described with reference to the previous figures. - At 502, a
frame 106, of a series offrames 106, is processed for rendering on a display panel(s) 102 of aHMD 100. At this point in time, the panel mask(s) 110 that is to be rendered on the display panel(s) 102 along with theframe 106 may have size values associated with individual portions of the panel mask(s) 110 set to existing size values that dictate whether those portions of the panel mask(s) 110 are in a fully contracted state, a fully expanded state, or somewhere in between. When the panel mask(s) 110 has not previously been expanded to an increased size, the existing size values for each portion may be set to a minimum size value (e.g., a size value of zero) corresponding to a fully contracted state. - At 504, logic of the
HMD 100 may determine whether a most recently-renderedframe 106, of the series offrames 106, is a re-projected frame 106(R) that was derived from pixel data associated with a previously-rendered actual frame 106(A) that was received from an application. Said another way, the logic of theHMD 100 determines whether re-projection is occurring at present. If, at 504, it is determined that the series offrames 106 are being rendered on the display panel(s) 102 of theHMD 100 using re-projection, theprocess 500 may follow the “YES” route fromblock 504 to block 506. - At 506, logic of the
HMD 100 may determine, based at least in part on rotation data provided by a head tracking module of theHMD 100, target size values for individual portions of thepanel mask 110. For example, four target size values may be determined, based onHMD 100 rotation, clamped 0.0 to 1.0 per direction. That is, a first target size value between 0 and 1 (inclusive) may be determined for a left portion of thepanel mask 110, a second target size value between 0 and 1 (inclusive) may be determined for a top portion of thepanel mask 110, a third target size value between 0 and 1 (inclusive) may be determined for a right portion of thepanel mask 110, and a fourth target size value between 0 and 1 (inclusive) may be determined for a bottom portion of thepanel mask 110. This may be done for eachpanel mask 110, which means that a total of eight target size values may be determined atblock 506 for left and right panel masks 110(L) and 110(R). In general, the direction and the amount ofHMD 100 rotation dictates the portion of thepanel mask 110 and the target size value, respectively, for that portion of thepanel mask 110. - As shown by
sub-block 508, the logic of theHMD 100 may determine, based on the rotation data provided by the head tracking module of theHMD 100, an amount of rotation of theHMD 100 per direction (e.g., amounts of rotation in left, right, up, and down directions) over a period of time since a renderedframe 106, of the series offrames 106, was last rendered. Any suitable manner of determining an amount of rotation in any unit of measurement may be utilized for the determination atblock 508. For instance, a number of degrees of rotation between frames can be determined as the amount of rotation. In some embodiments, an amount of rotation determined atblock 508 may be in the form of angular velocity. Directionality of the HMD's 100 rotation may be determined with reference to positive or negative horizontal and vertical directions, or any other suitable form of directional measurement, and the directionality of the HMD's 100 rotation may be broken down into component directions (e.g., left, right, up, and/or down directions in the X-Y plane). Accordingly, the logic atblock 506 may include determining a portion(s), among multiple portions, of the panel mask(s) 110 that corresponds to the direction in which theHMD 100 has rotated. In general, the greater the amount of rotation of theHMD 100, the greater the target size value so that more of theimage area 108 on thedisplay panel 102 is covered by thepanel mask 110 for a greater amount of rotation. In this sense, there may be a threshold amount of rotation that corresponds to a maximum size value for the portion of thepanel mask 110, and if the amount of rotation of theHMD 100 meets or exceeds this threshold amount, the maximum size value may be selected as the target size value atblock 506. If the amount of rotation of theHMD 100 is less than this threshold amount, the target size value is determined as something less than the maximum size value. In some embodiments, the target size value may be determined based on the relative differences of the re-projection transforms used for theprevious frame 106 and thecurrent frame 106. That is, instead of, or in addition to, determining the amount of rotation of theHMD 100, a relative difference between the re-projection transforms used during theprevious frame 106 and thecurrent frame 106 can be used to drive the calculation of the target size value based on the notion that the greater the difference between the re-projection transforms, the more of the display panel exhibits the unwanted visual artifacts (e.g., strobing flash between black pixels and illuminated/valid pixels). - If, at 504, it is determined that the series of
frames 106 are not being rendered on the display panel(s) 102 of theHMD 100 using re-projection, theprocess 500 may follow the “NO” route fromblock 504 to block 510. After a previous iteration, this determination atblock 504 may be a determination that re-projection has ceased. For instance, after rendering aframe 106 and proceeding to process a next frame at 502, the logic of theHMD 100 may determine that the series offrames 106 are being rendered without re-projection (e.g., without any re-projected frames 106(R)), indicating that the use of re-projection has ceased. Said another way, one or more future frames to be rendered on thedisplay panel 102 may not include any re-projected frames 102(R), indicating that the use of re-projection has ceased, or is otherwise not occurring at present. This may occur when the application hits frame rate after previously failing to hit frame rate. - At 510, logic of the
HMD 100 may set the target size values for individual portions of thepanel mask 110 to zero (or a minimum size value). For instance, four target size values corresponding to left, top, right, and bottom portions of thepanel mask 110 may be set to zero atblock 510. - At 512, following either block 506/508 or block 510, logic of the
HMD 100 may determine, for each individual portion of thepanel mask 110, whether the target size value is greater than an existing size value for the portion of thepanel mask 110. If it is determined, atblock 512, that the target size value is greater than the existing size value, theprocess 500 may follow the “YES” route fromblock 512 to block 514. - At 514, logic of the
HMD 100 may adjust (or change) a size value associated with the portion of thepanel mask 110 from the existing size value to the target size value to increase a size of the portion of thepanel mask 110 from a current size to an increased size. As indicated, this adjustment to increase the size may be based on the target size value being greater than the existing size value, indicating that the portion of thepanel mask 110 is to be brought inward to cover more of theimage area 108. In some embodiments, the size adjustment atblock 514 is temporally smoothed to mitigate a sudden field of view change. - At 516, logic of the
HMD 100 may render theframe 106 on the display panel(s) 102 along with the panel mask(s) 110 rendered at a periphery of the display panel(s) 102, wherein at least the portion of the panel mask is rendered at a size in accordance with the target size value, and atblock 518, thenext frame 106, in the series offrames 106, may be processed by iterating fromblock 502 for the next frame. Followingblock 514, thepanel mask 110 may be rendered atblock 516 at an increased size due to the increase from the existing size value to the target size value. It is also to be appreciated that, for aHMD 100 with a pair of display panels (e.g., the left display panel 102(L) and the right display panel 102(R)), the algorithm ofFIG. 5 may be done independently for each panel mask 110(L) and 110(R). In practice, the same portions of each mask may be adjusted in the same, or a similar, manner in response toparticular HMD 100 rotation during re-projection, such as when the leading edge portion of thepanel mask 110 on eachdisplay panel 102 is expanded inward toward the center of eachdisplay panel 102, as depicted in the examples ofFIGS. 1A and 1B . Furthermore, theframe 106 for which thepanel mask 110 is rendered in accordance with newly determined target size values may be an actual frame 106(A) or a re-projected frame 106(R). In at least one example, the determination, atblock 504, that re-projection is occurring is based on a previously-rendered re-projected frame 106(R). Accordingly, the subsequent frame may be an actual frame 106(A) received from the application, in which case, thepanel mask 110 is rendered atblock 516 in accordance with the calculated target size values along with the actual frame 106(A). Said another way, when re-projection is first detected, thefirst frame 106 that is rendered with apanel mask 110 that has been adjusted in size may be the next frame, which can be an actual frame 106(A) received from the application. In some embodiments, however, such as when multiple re-projected frames 106(R) are rendered between sequential actual frames 106(A) (e.g., when the application is outputting frames at, say, 30 frames per second, and two re-projected frames 106(R) are rendered between every actual frame 106(A)), thefirst frame 106 that is rendered with the increasedsize panel mask 110 may be the next re-projected frame 106(R). - Returning with reference to block 512, if it is determined, at
block 512, that the target size value is not greater than the existing size value, theprocess 500 may follow the “NO” route fromblock 512 to block 520. At 520, logic of theHMD 100 may determine whether the target size value (for an individual portion of the panel mask 110) is less than the existing size value. If the answer atblock 520 is “no”, theprocess 500 may follow the “NO” route fromblock 520 to block 522, based on the determination that the target size value is equal to the existing size value. At 522, no adjustment is made to the size value for the given portion of thepanel mask 110, and the frame is rendered atblock 516 without any adjustment to the size of the portion of thepanel mask 110. In other words, if it is determined that thepanel mask 110 is already covering what it is supposed to be covering, then no adjustment is made. If, on the other hand, it is determined, atblock 520, that the target size value for the portion of thepanel mask 110 is less than the existing size value, theprocess 500 may follow the “YES” route fromblock 520 to block 524. - At 524, logic of the
HMD 100 may determine whether a predetermined number of frames 106 (e.g., N frames, where N is equal to 5, 10, etc.) have been consecutively rendered without increasing a size of the relevant portion of thepanel mask 110. If there has been an increase in the size of the relevant portion of thepanel mask 110 over the course of the last Nconsecutive frames 106, theprocess 500 may follow the “NO” route fromblock 524 to block 522, where no adjustment is made to that portion of thepanel mask 110, and to block 516 where the frame is rendered without any adjustment to the size of the relevant portion of thepanel mask 110. This may be based on the notion that it may be beneficial to be somewhat confident that re-projection has ceased before contracting thepanel mask 110 to uncover a previously covered portion of theimage area 108. This confidence can be based on a particular number of consecutive frames 106(A) being rendered without increasing the size of the relevant portion of thepanel mask 110. Accordingly, if there hasn't been any increase to the size of the relevant portion of thepanel mask 110 over the course of the last Nconsecutive frames 106, theprocess 500 may follow the “YES” route fromblock 524 to block 526. - At 526, the size value may be decremented (e.g., incrementally adjusted downward) towards a lesser target value (e.g., the target size value, or otherwise to a minimum size value of “0”, corresponding to a fully contracted state of the panel mask 110). This may allow for decrementing the size of the portion of the portion of the
panel mask 110 over a time period (e.g., a period of roughly two seconds) from the existing size value to the target size value that is less than the existing size value, causing the portion of thepanel mask 110 to decrease in size over the period of time from the current size to a lesser size. Thus, over the course of multiple frames, the decrementing atblock 526 will eventually cause a previously expanded portion of thepanel mask 110 to decrease in size. As head rotation slows and/or re-projection ceases, the previously expanded portion of thepanel mask 110 may eventually decrease to a minimum size over multiple iterations of theprocess 500. This enables a controlled rate of contraction of thepanel mask 110 at a rate that is unnoticeable to the human eye. In this manner, the contraction of thepanel mask 110 may not be a distraction to theuser 104. In some embodiments, the decrementing operation atblock 526 is a slow interpolation step that decrements the existing size value to a lesser target value by a relatively small amount (e.g., a per-frame lerp value of 0.005). In other words, the logic may be configured to avoid contracting portions of thepanel mask 110 faster than a maximum rate of contraction, meaning that if the user's 104 head rotation slows suddenly by a large amount, instead of decreasing the size of the portion of thepanel mask 110 all the way down to the determined target size value, the existing size value can, instead, be decremented to an intermediate size value that is greater than the target size value and less than the existing size value. In some embodiments, if the target size value is between the lesser size value and the existing size value, the decrementing atblock 526 decrements to the target size value. After decrementing the existing size value (e.g., making the existing size value a particular percentage (e.g., 5%) smaller), theprocess 500 may proceed fromblock 526 to block 516 where theframe 106 is rendered on thedisplay panel 102 along with the panel mask, wherein the portion of thepanel mask 110 that was decremented in size is rendered at a decreased size, relative to a previous size of the portion of thepanel mask 110. - It can be appreciated that, upon subsequent iterations of the
process 500, it may be determined, atblock 504, that re-projection is still being used for rendering the series offrames 106, or that re-projection has ceased, which dictates whether the individual portions of thepanel mask 110 are to be dynamically increased, decreased, or left unchanged relative to a previous size adjustment. In general, the appropriate portions of thepanel mask 110 increase in size during re-projection coupled withmore HMD 100 rotation, and whenHMD 100 rotation slows and/or re-projection ceases, thepanel mask 110 portions may decrease in size to revealmore image area 108 on thedisplay panel 102. -
FIG. 6 illustrates example components of aHMD 600, such as a VR headset, according to the embodiments disclosed herein may be embedded. TheHMD 600 may be the same as, or similar to, theHMD 100 referenced in the previous figures, and, therefore, the components of theHMD 600 shown inFIG. 6 may be implemented in theHMD 100. TheHMD 600 may be implemented as a standalone device that is to be worn by a user 104 (e.g., on a head of the user 104). In some embodiments, theHMD 600 may be head-mountable, such as by allowing auser 104 to secure theHMD 600 on his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of auser 104. In some embodiments, theHMD 600 comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display(s). As such, the terms “wearable device”, “wearable electronic device”, “VR headset”, “AR headset”, and “head-mounted display (HMD)” may be used interchangeably herein to refer to thedevice 600 ofFIG. 6 . However, it is to be appreciated that these types of devices are merely example of aHMD 600, and it is to be appreciated that theHMD 600 may be implemented in a variety of other form factors. - In the illustrated implementation, the
HMD 600 includes one ormore processors 602 and memory 604 (e.g., computer-readable media 604). In some implementations, the processors(s) 602 may include a central processing unit (CPU), a graphics processing unit (GPU), both CPU and GPU, a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) 602 may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems. - The
memory 604 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. - The
memory 604 may be implemented as computer-readable storage media (“CRSM”), which may be any available physical media accessible by the processor(s) 602 to execute instructions stored on thememory 602. In one basic implementation, CRSM may include random access memory (“RAM”) and Flash memory. In other implementations, CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) 602. - In general, the
HMD 600 may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable media 604 is shown as including various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) 602 for carrying out the techniques, functionality, and/or operations described herein. A few example functional modules are shown as stored in the computer-readable media 604 and executable on the processor(s) 602, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic. - An
operating system module 606 may be configured to manage hardware within and coupled to theHMD 600 for the benefit of other modules. In addition, in some instances theHMD 600 may include one ormore applications 608 stored in thememory 604 or otherwise accessible to theHMD 600. In this implementation, the application(s) 608 includes agaming application 610. However, theHMD 600 may include any number or type of applications and is not limited to the specific example shown here. Thegaming application 610 may be configured to initiate gameplay of a video-based, interactive game (e.g., a VR game) that is playable by theuser 104, and to output frames (e.g., actual frames 106(A)) to be rendered on the display panels of theHMD 100. Acompositor 611 may be configured to render, and to dynamically adjust, apanel mask 110 along with each frame as a late-stage rendering operation. Accordingly, thecompositor 611, along with associated logic of theHMD 100, may perform the dynamic panel mask adjustment techniques disclosed herein. - Generally, the
HMD 600 hasinput devices 612 andoutput devices 614. Theinput devices 612 may include control buttons. In some implementations, one or more microphones may function asinput devices 612 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function asinput devices 612 to receive gestural input, such as a hand and/or head motion of theuser 104. In some embodiments,additional input devices 612 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like. In other embodiments, theHMD 600 may omit a keyboard, keypad, or other similar forms of mechanical input. Instead, theHMD 600 may be implemented relatively simplistic forms ofinput device 612, a network interface (wireless or wire-based), power, and processing/memory capabilities. For example, a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that theHMD 600 can thereafter be used. In one implementation, the input device(s) 612 may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons. - The
output devices 614 may include a display(s) 616, which may be the same as, or similar to, the display panel(s) 102 described with reference to the previous figures. Theoutput devices 614 may further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like. There may also be a simple light element (e.g., LED) to indicate a state such as, for example, when power is on. - The
HMD 600 may further include awireless unit 618 coupled to anantenna 620 to facilitate a wireless connection to a network. Thewireless unit 618 may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that theHMD 600 may further include physical ports to facilitate a wired connection to a network, a connected peripheral device (including a PC, game console, etc.), or a plug-in network device that communicates with other wireless networks. - The
HMD 600 may further includeoptical subsystem 622 that directs light from the electronic display(s) 616 to a user's eye(s) using one or more optical elements. Theoptical subsystem 622 may include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth. in some embodiments, one or more optical elements inoptical subsystem 622 may have one or more coatings, such as anti-reflective coatings. Magnification of the image light byoptical subsystem 622 allows electronic display(s) 616 to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., 120-150 degrees diagonal), and in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., about 40 degrees FOV).Optical subsystem 622 may be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to electronic display(s) 616 for display is pre-distorted, andoptical subsystem 622 corrects the distortion when it receives image light from electronic display(s) 616 generated based on the content. - The
HMD 600 may further include one ormore sensors 624, such as sensors used to generate motion, position, and orientation data. Thesesensors 624 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors, Thesensors 624 may also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data. TheHMD 600 may include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of theHMD 600. - In an example, the sensor(s) 624 may include an inertial measurement unit (IMU) 626.
IMU 626 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated withIMU 626, or sonic combination thereof Based on the measurement signals such motion-based sensors, such as theIMU 626, may generate calibration data indicating an estimated position ofHMD 600 relative to an initial position ofHMD 600. For example, multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll).IMU 626 can, for example, rapidly sample the measurement signals and calculate the estimated position ofHMD 600 from the sampled data. For example,IMU 626 may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point onHMD 600. The reference point is a point that may be used to describe the position of wearable device 702. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD 600 (e.g., a center of the IMU 626), Alternatively,IMU 626 provides the sampled measurement signals to an external console (or other computing device), which determines the calibration data. - The
sensors 624 may operate at relatively high frequencies in order to provide sensor data at a high rate. For example, sensor data may be generated at a rate of 1000 Hz (or 1 sensor reading every 1 millisecond). In this way, one thousand readings are taken per second. When sensors generate this much data at this rate (or at a greater rate), the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milliseconds. - As mentioned, in some embodiments, the
sensors 624 may include light sensors that are sensitive to light emitted by base stations in the environment of theHMD 600 for purposes of tracking position and/or orientation, pose, etc., of theHMD 600 in 3D space. The calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by thesensors 624. - The
HMD 600 may further include aneve tracking module 628. A camera or other optical sensor insideHMD 600 may capture image information of a user's eyes, andeye tracking module 628 may use the captured information to determine interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to HMD 600 (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted withinHMD 600 and reflected from each eye. The reflected light is received or detected by a camera of theHMD 600 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of auser 104 can be used byeye tracking module 628. Accordingly,eye tracking module 628 may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of auser 104 to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking). For example,eye tracking module 628 may integrate information from past measurements, measurements identifying a position of a user's 104 head, and 3D information describing a scene presented by electronic display(s) 616. Thus, information for the position and orientation of the user's 104 eyes is used to determine the gaze point in a virtual scene presented byHMD 600 where theuser 104 is looking. - The
HMD 600 may further include ahead tracking module 630. Thehead tracking module 630 may leverage one or more of thesensor 624 to track head motion, including head rotation, of theuser 104, as described above. For example, thehead tracking module 630 can track up to six degrees of freedom of the HMD 600 (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at everyframe 106 of a series offrames 106 so that theapplication 608 can determine how to render a scene in the next frame 106 (even for re-projected frames 106(R)) in accordance with the head position and orientation. In some embodiments, thehead tracking module 630 is configured to predict a future position and/or orientation of theHMD 600 based on current and/or past data. This is because the application is asked to render aframe 106 before theuser 104 actually sees the light (and, hence, the image) on the display(s) 616. Accordingly, anext frame 106 can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time, such as roughly 25-30 milliseconds (ms) prior to rendering theframe 106. Rotation data provided by thehead tracking module 630 can be used to determine both direction ofHMD 600 rotation, and amount ofHMD 600 rotation in any suitable unit of measurement. For example, rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down. Amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine a rate of rotation of theHMD 600. - Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.
Claims (20)
Priority Applications (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/033,162 US10520739B1 (en) | 2018-07-11 | 2018-07-11 | Dynamic panel masking |
EP19833668.7A EP3818410A4 (en) | 2018-07-11 | 2019-07-11 | Dynamic panel masking |
PCT/US2019/041432 WO2020014500A1 (en) | 2018-07-11 | 2019-07-11 | Dynamic panel masking |
KR1020217003833A KR20210029244A (en) | 2018-07-11 | 2019-07-11 | Dynamic panel masking |
JP2021500445A JP2021531493A (en) | 2018-07-11 | 2019-07-11 | Dynamic panel masking |
CN201980045960.0A CN112384843B (en) | 2018-07-11 | 2019-07-11 | Dynamic panel mask |
US16/712,370 US10948730B2 (en) | 2018-07-11 | 2019-12-12 | Dynamic panel masking |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/033,162 US10520739B1 (en) | 2018-07-11 | 2018-07-11 | Dynamic panel masking |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/712,370 Continuation US10948730B2 (en) | 2018-07-11 | 2019-12-12 | Dynamic panel masking |
Publications (2)
Publication Number | Publication Date |
---|---|
US10520739B1 US10520739B1 (en) | 2019-12-31 |
US20200018978A1 true US20200018978A1 (en) | 2020-01-16 |
Family
ID=69057602
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/033,162 Active US10520739B1 (en) | 2018-07-11 | 2018-07-11 | Dynamic panel masking |
US16/712,370 Active US10948730B2 (en) | 2018-07-11 | 2019-12-12 | Dynamic panel masking |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/712,370 Active US10948730B2 (en) | 2018-07-11 | 2019-12-12 | Dynamic panel masking |
Country Status (6)
Country | Link |
---|---|
US (2) | US10520739B1 (en) |
EP (1) | EP3818410A4 (en) |
JP (1) | JP2021531493A (en) |
KR (1) | KR20210029244A (en) |
CN (1) | CN112384843B (en) |
WO (1) | WO2020014500A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200225473A1 (en) * | 2019-01-14 | 2020-07-16 | Valve Corporation | Dynamic render time targeting based on eye tracking |
US11514875B1 (en) * | 2021-07-07 | 2022-11-29 | Htc Corporation | Method for mitigating dizziness, electronic device, and computer readable medium |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10962780B2 (en) * | 2015-10-26 | 2021-03-30 | Microsoft Technology Licensing, Llc | Remote rendering for virtual images |
EP3577896A4 (en) * | 2017-02-03 | 2020-11-25 | Warner Bros. Entertainment Inc. | Rendering extended video in virtual reality |
US10520739B1 (en) * | 2018-07-11 | 2019-12-31 | Valve Corporation | Dynamic panel masking |
US11741918B1 (en) * | 2021-02-22 | 2023-08-29 | Apple Inc. | Display with a vignetting mask |
WO2023151067A1 (en) * | 2022-02-14 | 2023-08-17 | Qualcomm Incorporated | Display mask layer generation and runtime adjustment |
Family Cites Families (37)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7310072B2 (en) | 1993-10-22 | 2007-12-18 | Kopin Corporation | Portable communication display device |
WO2002073955A1 (en) * | 2001-03-13 | 2002-09-19 | Canon Kabushiki Kaisha | Image processing apparatus, image processing method, studio apparatus, storage medium, and program |
JP2008096868A (en) * | 2006-10-16 | 2008-04-24 | Sony Corp | Imaging display device, and imaging display method |
US7982750B2 (en) | 2008-05-01 | 2011-07-19 | Yiling Xie | Test-wearing image producing method for personal products |
US8957835B2 (en) | 2008-09-30 | 2015-02-17 | Apple Inc. | Head-mounted display apparatus for retaining a portable electronic device with display |
US20130278631A1 (en) * | 2010-02-28 | 2013-10-24 | Osterhout Group, Inc. | 3d positioning of augmented reality information |
US8964298B2 (en) * | 2010-02-28 | 2015-02-24 | Microsoft Corporation | Video display modification based on sensor input for a see-through near-to-eye display |
US8941559B2 (en) | 2010-09-21 | 2015-01-27 | Microsoft Corporation | Opacity filter for display device |
US10585472B2 (en) * | 2011-08-12 | 2020-03-10 | Sony Interactive Entertainment Inc. | Wireless head mounted display with differential rendering and sound localization |
JP5953963B2 (en) * | 2012-06-13 | 2016-07-20 | ソニー株式会社 | Head-mounted image display device |
US9996150B2 (en) * | 2012-12-19 | 2018-06-12 | Qualcomm Incorporated | Enabling augmented reality using eye gaze tracking |
GB201305402D0 (en) * | 2013-03-25 | 2013-05-08 | Sony Comp Entertainment Europe | Head mountable display |
US9443355B2 (en) * | 2013-06-28 | 2016-09-13 | Microsoft Technology Licensing, Llc | Reprojection OLED display for augmented reality experiences |
US10134189B2 (en) * | 2014-01-23 | 2018-11-20 | Sony Corporation | Image display device and image display method |
GB2523555B (en) * | 2014-02-26 | 2020-03-25 | Sony Interactive Entertainment Europe Ltd | Image encoding and display |
GB2523740B (en) * | 2014-02-26 | 2020-10-14 | Sony Interactive Entertainment Inc | Image encoding and display |
US9995933B2 (en) | 2014-06-24 | 2018-06-12 | Microsoft Technology Licensing, Llc | Display devices with transmittance compensation mask |
US20170186236A1 (en) * | 2014-07-22 | 2017-06-29 | Sony Corporation | Image display device, image display method, and computer program |
WO2016013272A1 (en) * | 2014-07-23 | 2016-01-28 | ソニー株式会社 | Information processing device, information processing method and image display system |
GB2528699B (en) * | 2014-07-29 | 2017-05-03 | Sony Computer Entertainment Europe Ltd | Image processing |
GB2534847A (en) | 2015-01-28 | 2016-08-10 | Sony Computer Entertainment Europe Ltd | Display |
US9874932B2 (en) * | 2015-04-09 | 2018-01-23 | Microsoft Technology Licensing, Llc | Avoidance of color breakup in late-stage re-projection |
JP6505556B2 (en) * | 2015-09-07 | 2019-04-24 | 株式会社ソニー・インタラクティブエンタテインメント | INFORMATION PROCESSING APPARATUS AND IMAGE GENERATION METHOD |
JP6511386B2 (en) * | 2015-11-20 | 2019-05-15 | 株式会社ソニー・インタラクティブエンタテインメント | INFORMATION PROCESSING APPARATUS AND IMAGE GENERATION METHOD |
US9961297B2 (en) * | 2015-12-26 | 2018-05-01 | Intel Corporation | Method and system of rotation of video frames for displaying a video |
US10089725B2 (en) | 2016-02-09 | 2018-10-02 | Google Llc | Electronic display stabilization at a graphics processing unit |
US10297077B1 (en) * | 2016-03-02 | 2019-05-21 | Valve Corporation | Hidden area stencil mesh rendering systems and methods |
US10068553B2 (en) * | 2016-09-12 | 2018-09-04 | Intel Corporation | Enhanced rendering by a wearable display attached to a tethered computer |
US10212428B2 (en) * | 2017-01-11 | 2019-02-19 | Microsoft Technology Licensing, Llc | Reprojecting holographic video to enhance streaming bandwidth/quality |
US10628994B2 (en) * | 2017-03-07 | 2020-04-21 | Google Llc | Reducing visually induced motion sickness in head mounted display systems |
US10255891B2 (en) * | 2017-04-12 | 2019-04-09 | Microsoft Technology Licensing, Llc | No miss cache structure for real-time image transformations with multiple LSR processing engines |
US10831028B2 (en) * | 2017-06-14 | 2020-11-10 | Sony Interactive Entertainment Inc. | Tracking using retroreflectors mounted on a head-mounted display |
US10520739B1 (en) * | 2018-07-11 | 2019-12-31 | Valve Corporation | Dynamic panel masking |
US10600236B2 (en) * | 2018-08-06 | 2020-03-24 | Valve Corporation | Prediction and throttling adjustments based on application rendering performance |
US10692420B2 (en) * | 2018-09-13 | 2020-06-23 | Arm Limited | Data processing systems |
US10733783B2 (en) * | 2018-10-09 | 2020-08-04 | Valve Corporation | Motion smoothing for re-projected frames |
US10802287B2 (en) * | 2019-01-14 | 2020-10-13 | Valve Corporation | Dynamic render time targeting based on eye tracking |
-
2018
- 2018-07-11 US US16/033,162 patent/US10520739B1/en active Active
-
2019
- 2019-07-11 WO PCT/US2019/041432 patent/WO2020014500A1/en unknown
- 2019-07-11 JP JP2021500445A patent/JP2021531493A/en active Pending
- 2019-07-11 EP EP19833668.7A patent/EP3818410A4/en active Pending
- 2019-07-11 KR KR1020217003833A patent/KR20210029244A/en unknown
- 2019-07-11 CN CN201980045960.0A patent/CN112384843B/en active Active
- 2019-12-12 US US16/712,370 patent/US10948730B2/en active Active
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200225473A1 (en) * | 2019-01-14 | 2020-07-16 | Valve Corporation | Dynamic render time targeting based on eye tracking |
US10802287B2 (en) * | 2019-01-14 | 2020-10-13 | Valve Corporation | Dynamic render time targeting based on eye tracking |
US11514875B1 (en) * | 2021-07-07 | 2022-11-29 | Htc Corporation | Method for mitigating dizziness, electronic device, and computer readable medium |
Also Published As
Publication number | Publication date |
---|---|
US10948730B2 (en) | 2021-03-16 |
US20200117015A1 (en) | 2020-04-16 |
CN112384843B (en) | 2023-06-06 |
JP2021531493A (en) | 2021-11-18 |
EP3818410A4 (en) | 2022-04-06 |
KR20210029244A (en) | 2021-03-15 |
US10520739B1 (en) | 2019-12-31 |
CN112384843A (en) | 2021-02-19 |
EP3818410A1 (en) | 2021-05-12 |
WO2020014500A1 (en) | 2020-01-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10948730B2 (en) | Dynamic panel masking | |
US10733783B2 (en) | Motion smoothing for re-projected frames | |
US11151776B2 (en) | Prediction and throttling adjustments based on application rendering performance | |
US10802287B2 (en) | Dynamic render time targeting based on eye tracking | |
US11343486B2 (en) | Counterrotation of display panels and/or virtual cameras in a HMD | |
US11749141B2 (en) | Information processing apparatus, information processing method, and recording medium | |
US11119313B1 (en) | Imaging systems and methods for expanding field of view of captured images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: VALVE CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:VLACHOS, ALEX;REEL/FRAME:046325/0458 Effective date: 20180710 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY Year of fee payment: 4 |