Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/04—Context-preserving transformations, e.g. by using an importance map
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/361—Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/013—Eye tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4092—Image resolution transcoding, e.g. by using client-server architectures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/60—Network streaming of media packets
  • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/60—Network streaming of media packets
  • H04L65/70—Media network packetisation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/60—Network streaming of media packets
  • H04L65/75—Media network packet handling
  • H04L65/764—Media network packet handling at the destination 
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/80—Responding to QoS
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L67/00—Network arrangements or protocols for supporting network services or applications
  • H04L67/01—Protocols
  • H04L67/131—Protocols for games, networked simulations or virtual reality
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/156—Mixing image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/167—Synchronising or controlling image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/194—Transmission of image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/366—Image reproducers using viewer tracking
  • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/398—Synchronisation thereof; Control thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/162—User input
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/164—Feedback from the receiver or from the transmission channel
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/167—Position within a video image, e.g. region of interest [ROI]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
  • H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
  • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
  • H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
  • H04N21/2365—Multiplexing of several video streams
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
  • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
  • H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
  • H04N21/4347—Demultiplexing of several video streams
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
  • H04N21/44213—Monitoring of end-user related data
  • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/81—Monomedia components thereof
  • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video

Definitions

• FIG. 1 is a block diagram of one implementation of a system.
• FIG. 2 is a block diagram of one implementation of a transmitter.
• FIG. 4 is a block diagram of one implementation of a receiver.
• FIG. 5 is a generalized flow diagram illustrating one implementation of a method for adjusting the phase timing of a foveal region stream.
• FIG. 6 is a generalized flow diagram illustrating one implementation of a method for increasing the number of intermediate resolution images that are rendered in response to an eye saccade being detected.
• FIG. 7 is a diagram of transported images and a final image in accordance with one implementation.
• FIG. 8 is a timing diagram of a regular multi-stream display transport in accordance with one implementation.
• FIG. 9 is a timing diagram of a phase-shifted multi-stream display transport in accordance with one implementation.
• phase timing of the foveal region stream(s) being sent over the transport layer is adjusted with respect to the background stream to correspond to the location of the foveal region within the overall image. This helps to reduce the amount of buffering needed at the receiver for blending the images together to create the composite image to be driven to the display.
• system 100 includes transmitter 105, channel 110, receiver 115, and head-mounted display (HMD) 120. It is noted that in other implementations, system 100 can include other components than are shown in FIG. 1.
• channel 110 is a direct wired connection between transmitter 105 and receiver 115.
• channel 110 is representative of a network connection between transmitter 105 and receiver 115. Any type and number of networks can be employed depending on the implementation to provide the connection between transmitter 105 and receiver 115.
• transmitter 105 is part of a cloud-service provider in one particular implementation.
• channel 110 represents a wireless connection between transmitter 105 and receiver 115.
• receiver 115 When receiver 115 receives the multiple streams, receiver 115 extracts the individual images from the streams and then blends the images together to create a single frame which is then driven to HMD 120.
• receiver 115 is separate from HMD 120, and receiver 115 communicates with HMD 120 using a wired or wireless connection. In another implementation, receiver 115 is integrated within HMD 120.
• the scene being displayed on the right side 125R of HMD 120 includes a foveal region 130R while the scene being displayed on the left side 125L of HMD 120 includes a foveal region 130L.
• These foveal regions 130R and 130L are indicated by the circles within the expanded right side 125R and left side 125L, respectively, of HMD 120.
• the term “foveal region” is defined as the portion of a per-eye displayed half frame where the gaze of each eye is directed. In some cases, the “foveal region” is determined based at least in part on an eye-tracking sensor detecting the location within the half frame where the eye is pointing. Additionally, the term “foveal region” can also be referred to as a “focus region”.
• the locations of foveal regions 130R and 130L within the right and left half frames, respectively, are determined based on one or more eye-tracking sensors 140 within HMD 120.
• the eye tracking data is provided as feedback to transmitter 105 and optionally to the rendering source of the VR video.
• the eye tracking data feedback is generated at a frequency higher than the VR video frame rate, and transmitter 105 is able to access the feedback and update the encoded video stream on a per- frame basis.
• the eye tracking is not performed on HMD 120, but rather, the video and other sensor data is sent back to transmitter 105 for further processing to determine the eye’s position and movement.
• channel 110 supports a multi-stream transport layer to transfer higher-resolution foveal region(s) as well as a lower-resolution background image, using a separate stream for each region and a separate stream for the background image.
• the foveal and background images are transmitted at different refresh rates. For example, if the user has moved their eyes rapidly while the current scene in the VR or AR application is unchanged, then the foveal region is sent at a higher refresh rate than the background image to account for new saccade landing points.
• transmitter 105 sends mask metadata to receiver 115 specifying how the multiple regions should be combined to create a final, composite image for the frame. Receiver 115 blends the foveal regions with the background image based on the received mask metadata.
• the mask metadata is supplied as an alpha color channel. In another implementation, the mask metadata is determined analytically. In one implementation, transmitter 105 also sends metadata which specifies a background image scaling attribute to receiver 115. The background image scaling attribute indicates how much the background image should be scaled by receiver 115 before blending with the foveal region(s). Receiver 115 scales the background image based on this received scaling attribute.
• receiver 115 When receiver 115 receives the multiple streams, receiver 115 extracts the image from each stream. Then, receiver 115 superimposes the foveal image(s) on the background image to create the final image which will be driven to display 120. Phase shifting the transmission of the foveal image stream(s) in relation to the background image stream reduces the amount of buffering needed on receiver 115 to combine the foveal image(s) with the background image to create the final image.
• transmitter 200 includes at least foveated rendering unit 210, eye saccade prediction unit 215, and encoder 225.
• foveated rendering unit 210 is coupled to transmitter 200 rather than being integrated within transmitter 200. It is noted that transmitter 200 can also include other components which are not shown to avoid obscuring the figure. While foveated rendering unit 210, eye saccade prediction unit 215, and encoder 225 are shown as separate units in FIG. 2, it is noted that in other implementations, any two of these units can be combined together into a single unit or all three units can be combined together into a single unit.
• Each of rendering unit 210, eye saccade prediction unit 215, and encoder 225 is implemented using any suitable combination of hardware (e.g., contr l logic, processing unit) and/or software (e.g., program instructions executable by a processing unit).
• foveated rendering unit 210 generates rendered images 220 from graphics information (e.g., raw image data).
• rendered images 220 include two or more images which correspond to a single video frame of a video sequence. It is noted that the terms “image”, “frame”, and “video frame” can be used interchangeably herein.
• foveated rendering unit 210 receives foveal region information which specifies the foveal region(s) (e.g., foveal regions 130R and 130L of FIG. 1) within the frame.
• the foveal region information is provided to transmitter 200 from one or more eye-tracking sensors (e.g., sensor 140) in the HMD (e.g., HMD 120).
• Foveated rendering unit 210 uses the foveal region information to generate the foveal region image at a relatively higher pixel density than the other images of rendered images 220.
• each image of rendered images 220 is representative of a pair of images with one image for each eye view of the HMD.
• each image of rendered images 220 is representative of a pair of images that include a first rendered image for the right eye and a second rendered image for the left eye.
• each image of rendered images 220 includes left-eye and right-eye portions which are combined together into a single image.
• encoder 225 generates multiple encoded streams 235 for each frame from rendered images 220. Then, encoder 225 sends the multiple encoded streams 235 to a receiver for reconstruction, with each stream including a separately encoded image representation of a portion or the entirety of the frame.
• each stream includes an image at a different resolution from the resolutions of the other images of the other streams.
• the update rate of each stream is independent of the other streams. Accordingly, in one implementation, the foveal region stream has a higher update rate than the background image stream.
• the location and timing predictions are used to determine when to render the foveal region(s) of the next frame to be displayed to the user.
• saccadic omission prevents some visual input from being processed for a short period before the rapid eye movement, during the eye movement, and for a short period after the eye movement.
• the brain will start processing new visual data again after a short delay. Therefore, to prevent the user from having a negative viewing experience, the foveated rendering unit 210 will render the foveal region to appear shortly after the predicted landing location of the eye saccade.
• foveated rendering unit 200 will render the next frame so that the next frame is displayed to the user shortly after the moment when the eye saccade is predicted to end.
• rendering unit 210 generates a plurality of images of different resolutions with the data in the foveal region(s) to approximate acuity falloff of the eye.
• the rendering unit 210 will increase the number of images that are rendered per frame and adjust their shape (e.g. elongate image in the direction of saccade, expand the image to cover a larger image portion at the saccade landing point) for some amount of time once the eye saccade is predicted to end to correct for possible landing point prediction errors.
• rendering unit 210 during periods of time of negligible eye movement, generates two images per frame that will be blended together by the receiver.
• rendering unit 210 in response to an eye saccade being detected, generates a foveal region image at a first resolution, a first-intermediate-foveal region image at a second resolution, a second-intermediate - foveal region image at a third resolution, and a background image at a fourth resolution.
• FIG. 3 a timing diagram 300 of frame timing for rendering and driving frames to a VR or AR display in accordance with one implementation is shown.
• the boundaries of each frame period are determined by vertical synchronization (VSync) signals.
• VSync vertical synchronization
• the start of a frame period coincides with a VSync signal and the end of the frame period coincides with the next VSync signal.
• the first frame period shown in timing diagram 300 is bounded by VSync 305 and Vsync 310.
• an original foveal image stream is sent in a phase-shifted manner with respect to the background image stream over the display transport to the receiver.
• the phase shifting is performed so that the foveal image stream arrives at the receiver at the point in time when the foveal image will be blended with the background image.
• the receiver blends the foveal region with the background image and then a VR compositor drives the blended image to a display.
• the term “frame period” is defined as the reciprocal of the frame rate.
• frames are driven to a VR display at a fixed or variable frame rate, with the frame rate high enough to provide a positive user experience for the given VR application. It is noted that throughout this disclosure, when a technique is referred to as applicable to a VR application or VR display, it should be understood that the technique is also applicable to AR or other types of immersive applications.
• the timing of the Vsync for the next frame to be displayed can be determined.
• the VR system includes an eye saccade detection unit or an eye saccade prediction unit (e.g., eye saccade prediction unit 215 of FIG. 2) that detects and/or predicts an eye saccade during execution of the VR application.
• an eye saccade 325 has been detected and the prediction of the timing of eye saccade 325 is shown in relation to Vsync signals, the transport of the background image stream, and the transport of the original foveal image stream.
• the location of the foveal region will move, which will change the optimal timing for sending the foveal image stream on the display transport.
• a new foveal image is rendered to account for the predicted location of the foveal region when eye saccade 325 ends.
• the new foveal image stream 330 is sent at a new phase relationship with respect to the background image stream 335 to correspond to when the new foveal image will be blended with the background image.
• This adjustment in the phase of new foveal image stream 330 is performed if there is sufficient time to render and transmit new foveal image stream 330 prior to the next Vsync 315. If there is not enough time before Vsync 315, then a lower-resolution background image can be displayed to the user for the current frame, and then the appropriate phase relationship between the new foveal image stream and the background image stream will be applied to the subsequent frame.
• receiver 400 receives multiple encoded streams for the frames of a VR or AR application. After receiving the multiple encoded streams, stream demultiplexer 410 separates the encoded streams into the individual component streams. In one implementation, there are two streams in the encoded streams, with the two streams including a relatively high-resolution foveal image and a relatively low-resolution background image. In other implementations, receiver 400 receives more than two encoded streams. The individual component streams are separated out and stored in buffers 415A-N, which are representative of any number of buffers for storing pixel data of the different component streams.
• Decode unit 420 retrieves the separate component encoded images from buffers 415A- N and decodes the encoded images. Then, decode unit 420 conveys the decoded images 425 to display compositor 430. Display compositor 430 blends the decoded images 425 together to create a blended image 435 which is provided to display controller 440. In one implementation, decode unit 420 extracts blend mask 422 from the encoded streams and conveys blend mask 422 to display compositor 430. In this implementation, display compositor 430 uses the blend mask 422 to determine how to combine the multiple component images together to create blended image 435. Display controller 440 performs the final adjustments to blended image 435 to create final image 445 which is driven to the display. It is noted that final image 445 can be stored in a frame buffer or other location prior to being driven to the target display.
• FIG. 5 one implementation of a method 500 for adjusting the phase timing of a foveal region stream is shown.
• steps in this implementation and those of FIG. 6-7 are shown in sequential order.
• one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely.
• Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method 500.
• a transmitter receives an indication of a location of a foveal region within a frame (block 505).
• the foveal region information is sent to the transmitter via a sideband channel from one or more eye-tracking sensors.
• the transmitter adjusts a phase timing of foveal region image stream with respect to a background image stream to correspond to the location of the foveal region within the frame (block 510).
• the phase timing of when the foveal region image is sent over the transport layer is moved forward to a point which allows the entirety of the foveal region image to be received by the receiver at the point in time when the foveal region image will need to be blended with the background image.
• the phase timing of when the foveal region image is sent over the transport layer is adjusted to allow the foveal region image to be fully received by the receiver in time for blending.
• the transmitter sends, to a receiver, the foveal region image stream using the adjusted phase timing with respect to the background image stream (block 515).
• method 500 ends.
• the receiver is able to reduce the amount of buffering that is used when blending the foveal region image with the background image.
• a rendering unit renders a relatively high-resolution image for a foveal region and a relatively low-resolution image for a background of each frame of a VR application (block 605). If the rendering unit receives an indication that an eye saccade has been detected for the current frame being rendered (conditional block 610, “yes” leg), then the rendering unit terminates rendering of the existing foveal region and starts rendering a new foveal region at a new location if there is sufficient time available based on the new phase timing (block 615).
• the receiver will handle the situation when the foveated stream is aborted and a new foveated stream is restarted at a different phase in a different location of the frame.
• the receiver is capable of dealing with multiple foveated regions within a frame due to such a restart.
• the rendering unit increases the number of intermediate resolution images that are rendered for the current frame (block 620). For example, in one implementation, the rendering unit renders a high-resolution foveal region, a medium-high resolution region surrounding the foveal region, a medium-resolution region surrounding the medium-high resolution region, and a low-resolution background image. In other implementations, the rendering unit renders other numbers of regions (greater than two) with varying amounts of resolution. In another implementation, for systems that use non- uniform resolution rendering techniques, multiple foveal layers could be generated from a higher resolution image.
• condition block 610 “no” leg
• the rendering unit renders a first image for the foveal region and a second image for the background (block 625).
• a blended image is driven to the display for the current frame (block 630), method 600 returns to conditional block 610.
• FIG. 7 a diagram of transported images and a final image in accordance with one implementation is shown.
• a low-resolution background image 705 is shown on the top left of FIG. 7, and one example of a high-resolution foveated image 710 is shown on the bottom left of FIG. 7.
• these two images 705 and 710 are sent from the transmitter to the receiver, with the phase-timing of the corresponding image streams adjusted based on the location of foveal region within the overall image.
• the receiver then blends images 705 and 710 together to create final image 715. It should be understood that other numbers of images can be sent to the receiver to be blended together to create a final image in other implementations.
• the phase timing of each higher-resolution image stream that is sent on the transport layer will be adjusted with respect the background image stream based on the locations of the corresponding region within the overall image.
• FIG. 8 a timing diagram of a regular multi-stream display transport in accordance with one implementation is shown.
• the timing of images being sent over the transport layer is shown at the top of FIG. 8.
• background image 805A and foveal image 805B are sent in phase over the transport layer, with background image 805A and foveal image 805B corresponding to a single frame being rendered and intended for display at the receiver.
• background image 805A and foveal image 805B are interleaved as they are sent over the transport layer.
• the timing of the processing of images at the receiver is shown below the transport timing blocks.
• background image 805A and foveal image 805B are sent in-phase over the transport layer to the receiver, the receiver stores the images in buffers 810A and 810B, respectively.
• background image 805 A is scaled as indicated by block 815A.
• the blending of the foveal image 805B starts at the appropriate time with respect to the scaling of background image 805A.
• the blending is represented by block 815B.
• the final image 817 is generated as a result of the scaling 815A and blending 815B steps.
• a new background image 820A and a new foveal image 820B are being sent to the receiver and stored in buffers 825 A and 825B, respectively.
• Two pairs of buffers 810A-B and 825A-B are needed since the background image 805A is being blended with foveal image 805B at the same time that background image 820A and foveal image 820B are being received. This timing pattern can continue for subsequent images being sent from the transmitter and processed by the receiver.
• the transmitter can abort the transmission of the foveal image stream and restart a new foveal image stream at the correct phase location with respect to the background image stream.
• program instructions of a software application are used to implement the methods and/or mechanisms described herein.
• program instructions executable by a general or special purpose processor are contemplated.
• such program instructions can he represented by a high level programming language.
• the program instructions can be compiled from a high level programming language to a binary, intermediate, or other form.
• program instructions can be written that describe the behavior or design of hardware.
• Such program instructions can he represented by a high-level programming language, such as C.
• a hardware design language (HDL) such as Veriiog can be used.
• the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution.
• a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Networks & Wireless Communication (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Social Psychology (AREA)
• Databases & Information Systems (AREA)
• Controls And Circuits For Display Device (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Image Processing (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=74880843

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (12)

Citations (6)

Family Cites Families (26)

• 2019
  • 2019-09-19 US US16/575,804 patent/US11307655B2/en active Active
• 2020
  • 2020-09-21 CN CN202080064752.8A patent/CN114402613A/zh active Pending
  • 2020-09-21 EP EP20866101.7A patent/EP4032283B1/en active Active
  • 2020-09-21 WO PCT/IB2020/058783 patent/WO2021053644A1/en not_active Ceased
  • 2020-09-21 JP JP2022515065A patent/JP7477600B2/ja active Active
  • 2020-09-21 KR KR1020227011803A patent/KR102689029B1/ko active Active

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events